Icechunk data repository

[JustinElms]

Background

This enables the use of Icechunk data stores in the Navigator as the first step in replacing our current data indexing solution. Most of these changes are related to updating the configurations and tests. Only minor changes were needed to actually enable the use of Icechunk.

This approach only supports S3 and local file storage configurations. In production MinIo will be used to enable data access while the local option is only used for testing however users could configure their own Navigator deployments to use either.

Why did you take this approach?

Since Icechunk interfaces so well with xarray we only needed to add functions to access the repositories. Everything else in the dataflow should be business as usual.

Anything in particular that should be highlighted?

All code related to our current data indexing solution has been left intact but will likely be removed once we fully migrate to this approach.

Checks

• I ran unit tests.
• I've tested the relevant changes from a user POV.
• I've formatted my Python code using black ..