Test and document using GDAL virtual file system handlers

[emlys]

Fixes #441

• Add a few tests using vsicurl (testing against new test data included in this repo, so that we can reference it at the github hosted URL)
• Add a section to the documentation about using virtual file system handlers
• Update all docstrings to indicate that GDAL vsi paths are supported for inputs

[dcdenu4]

Thanks @emlys , I found a couple other places we might want to add the vsi blurb too. One I couldn't really comment on is under the def build_overviews functions in geoprocessing.py, line 4670.

When thinking about testing a few questions came to mind that I was curious if you'd come across or had thought about.

1. Where does GDAL save the needed downloaded data from URLs and how / when does GDAL garbage collect it? Is that something we need to be mindful of?
2. I'd love to have some more comprehensive testing that could help us spot issues further in advance, but I know we don't want to burden our GHA runners all the time. It might be nice to start thinking about having a more comprehensive test runner / suite that we can manually run every once in awhile or that runs periodically during off hours.

[dcdenu4]

This got me thinking that it'd be nice to document what test_data/raster.tif is and it's properties. Maybe a more descriptive name for that test file could help at the very least. But if I wanted to use it in another test function I wouldn't know what to expect for values.

[emlys]

Hmm, I'm not sure exactly how to handle that. I renamed the files to small_raster.tif and small_vector.gpkg so hopefully that helps a bit? But I don't want to be too specific e.g. small_raster_for_vsi_tests.tif would then not make sense if we did want to use it in another test.

[dcdenu4]

Would it be overkill to have a small README in the folder that describes the data? I'm mostly thinking that I don't know what the values are and therefore what I'd expect for some kind of output. Maybe we can address this in the future if we do more of this.

[dcdenu4]

In playing around with this, do you have a sense for where GDAL downloads the data it needs from a hosted file? Are we sure we're cleaning that up in our tests?

[emlys]

I don't know that it downloads it to anywhere - it may just be in memory. I wouldn't expect it to leave any extra files around afterward

[emlys]

Thanks @dcdenu4! Here are my thoughts -

Where does GDAL save the needed downloaded data from URLs and how / when does GDAL garbage collect it? Is that something we need to be mindful of?

In the docs I do not see any mention of downloaded data being permanently stored. I think it's safe to assume that this is abstracted away by the drivers. The documentation suggests that everything is cached and there is a reasonable limit on the cache size:

Partial downloads (requires the HTTP server to support random reading) are done with a 16 KB granularity by default. Starting with GDAL 2.3, the chunk size can be configured with the CPL_VSIL_CURL_CHUNK_SIZE configuration option, with a value in bytes. If the driver detects sequential reading, it will progressively increase the chunk size up to 128 times CPL_VSIL_CURL_CHUNK_SIZE (so 2 MB by default) to improve download performance.
In addition, a global least-recently-used cache of 16 MB shared among all downloaded content is used, and content in it may be reused after a file handle has been closed and reopen, during the life-time of the process or until VSICurlClearCache() is called. Starting with GDAL 2.3, the size of this global LRU cache can be modified by setting the configuration option CPL_VSIL_CURL_CACHE_SIZE (in bytes).

I'd love to have some more comprehensive testing that could help us spot issues further in advance, but I know we don't want to burden our GHA runners all the time. It might be nice to start thinking about having a more comprehensive test runner / suite that we can manually run every once in awhile or that runs periodically during off hours.

I think this could be addressed separately... I don't want to fall into the trap of testing GDAL functionality rather than pygeoprocessing functionality. VSI handlers are new to us, but not new to GDAL, so I think we can assume that they work as described and not need to write our own comprehensive tests using that feature.

I did also test locally running raster_map on a huge global dataset from the data hub, and it worked as expected and I observed that memory use did not climb during the run time.

[dcdenu4]

Thanks @emlys for taking the time to reply to my questions.