Add batch query to `get_entity`

[ohjakobsen]

If a user provides more than one entity to get_entity, the function will perform multiple API calls. The API supports comma separated lists of organisation numbers or organisation names. Batching the queries to the API will drastically reduce the time to collect multiple entities and will reduce load on the API-service.

[ohjakobsen]

Note that by using query params, the API will return status code 200 regardless of whether or not the call returns any records, e.g. a GET request to https://data.brreg.no/enhetsregisteret/api/enheter?organisasjonsnummer=123456789 will return an HTTP 200 with an empty result set.

[ohjakobsen]

Currently, get_entities will return a maximum of 20 entities per request, as pagination is not supported. Pagination will be added to this issue as well.

[ohjakobsen]

This is proving to be a bit more tricky than first thought. The logic for all functions in noAPI rely heavily on one function call equaling one API call and that multiple requests are handled with lapply. Adding batch queries at first seems to make the logic easier, as we could (in theory) drop the lapply and send multiple entities in the same request. However, we need to do some validation on the input data.

The maximum input to the API is 10,000 entities. However, the results are paged, and we should as a general rule keep the number well below 10,000. Since the API require query parameters with GET-requests, the URI length should be kept under 2000 characters, which will give a maximum of just shy of 200 entities per request. To complicate further, the function does not care about main or sub entities (with default values), but the API is split on the two types. With batch requests, we need to parse the results, extract all returned entities, and then resubmit the missing entities to the other API endpoint (the batch version always return a 200 status code, so we cannot rely on looking for a 404 to catch missing entities).

The consequence of this, is that for most use cases, the function will need to perform multiple API calls and parse the results in some shape or form. This complicates things if we want to keep the logic where the user can choose not to simplify the result vector, or the return the raw response from the API. I'm not sure if we are able to keep all arguments to the function without violating the logic behind them. At the same time, batch calls to the API will speed up data retrieval significantly and will also be nicer to the API. This functionality should therefore be added. The question now is how.

[Aeilert]

Thanks @ohjakobsen. I fully agree. Our main goal should be usability, so batch queries should certainly be added.

On the other hand I do kind of like that there is an overall consistency between the exported API functions. For simple queries the current implementation is fine. It is only when querying several hundred or thousand entities that the implentation starts to be a problem.

A solution could be to add a seperate method / function with a much simpler logic, e.g. that it is not possible to get the response in raw format and that we don't care about input and output lengths. Imagine something like get_entity(x, method = c('regular', 'batch'), ....), where all other arguments are ignored if the "batch" method is selected.

Pro:

• We keep the current logic and consistency with other functions
• Current workflow and tests can be kept, we just need add a seperate internal workflow

Con:

• The user needs to explicitly select batch queries
• Even more complexity behind the get_entity() function

[ohjakobsen]

I'm not a big fan of supporting two different modes. I can't think of a realistic use case where the user deliberately want to slow down performance, and add stress on the API at the same time. Batch processing should be the default. On the other hand, I do – for the same reasons as you – like the idea of internal consistency. I'm working on a solution, but we might end up with two different approaches to this API.

[Aeilert]

Sounds good. Looking forward to see the solution :)