Add catmandu fix `import` function

[TobiasNx]

# Replace the data in foo.bar with an external file or url
import(foo.bar, JSON, file: "http://foo.com/bar.json", data_path: data.*)

Use case would be building a lobid gnd url in fix, then fetching the JSON-Data as string and then processing it with from_json and only copying the prefLabel.

In order to not break the workflow if no data can be fetched, we also should introduce the maybe bind, see: #699

[dr0i]

Isn't this what can be achieved already with put_rdfmap or/and put_filemap ?

[blackwinter]

Just be aware of the limitations:

Use case would be building a lobid gnd url in fix, then fetching the JSON-Data as string and then processing it with from_json and only copying the prefLabel.

The transformation would then become dependent on the availability of the external API as well as your network connection; it may also become considerably slower as opposed to a single bulk download, or even a local file, with put_filemap.

In order to not break the workflow if no data can be fetched, we also should introduce the maybe bind, see: #699

This could lead to silent data corruption/incompleteness.

[TobiasNx]

The limit with regard to performance especially in scenarios of big transformations of a hugh number records seems obvious to me.

Good that you hint to the possible data corruption. To keep that in mind.

The use case came up in context of RPB, where the number of records that are transformed are limited. I knew that Catmandu has a function for that. So I opened the ticket for that.

Alternatively @fsteeg suggested a new reconciliation function in fix but this would have the same disadvantage.

[blackwinter]

Alternatively @fsteeg suggested a new reconciliation function in fix but this would have the same disadvantage.

If lobid-gnd had a CSV export similar to lobid-organisations, you could use it for a remote lookup map.

In Limetrans, we're creating a local lookup map from the JSON.

[fsteeg]

In Limetrans, we're creating a local lookup map from the JSON.

Ah, right, thanks! Given that our use case in RPB is already based on a local (incomplete) lookup file, this is probably what we should do too.