Set max batch size arcgis for pipeline to run

[pooleycodes]

Currently some arcgis pages falsely say their max page size, when the download runs with this batch size it causes the endpoint to timeout.

an example of this is found at this url:

https://services1.arcgis.com/ESMARspQHYMw9BZ9/arcgis/rest/services/Local_Authority_Districts_December_2024_Boundaries_UK_BFE/FeatureServer/0/

Technical Approach:

• This is primarily an issue with how they've configured their arcgis endpoint. They should've set a max_page_size that supports the size of the data being downloaded as we automatically check the arcgis feature layer endpoint for that propety but they still have it set to 2000.
• It's likely that they will not fix this on their end so we need to implement functionality on our side to set this parameter when collecting
• The collector class is here which is the class responsible for downloading data. Specifically the arcgis plugin is called in the fetch method. and the plugin file that is used for arcgis plugins is here
• to support this there is an empty unused parameters column on the endpoint CSV (example)
• we should allow a parameter to be passed as an argument to the arcgis plugin from the endpoint CSV. This will need several changes to the code. There is a suggestion of this below

to fix this it may be that the endpoint csv file has an optional batch size parameters such to be pulled in and set the max_page_size

Such as:
EsriDumper(url, fields=None, max_page_size=20)

Acceptance Criteria:

• to be able to manually set max page limit for endpoints using the arcgis plugin
• to collect from the example url above
• to error if incorrect parameters are provided from the endpoint CSV

[salabi]

I have added max_page_size to the ArcGIS plugin and propagated it into the collect.py code. However, after running several local tests, I’ve observed that the runtime is inconsistent and occasionally times out, even when max_page_size is set to 20. I am currently investigating this issue.
branch: https://github.com.mcas.ms/digital-land/digital-land-python/tree/argis_parameter_dev