Zarr vs LitData - Loading data effectively

[daniel-z-kaplan]

There are multiple methods of loading data for use with pathology.
Zarr and LitData are two of the ways we are currently considering this data storing.

This task is being deliberately left fairly open - we are trying to understand the best ways to process and load the data, for fast training.

Some things (non-exhaustive list) to consider are:

• limited disk space on the training cluster
• limited streaming speed
• potential limits on disk storing space

[PaulScotti]

Hoping someone can volunteer to help with this!

Relatedly, I came across this post which details someone else's approach to how to efficiently handle Zarr for fast throughout cloud-storage model training, which includes using xarray + xbatcher + Dask. Probably with their approach Zarr loading could be made way more efficient

https://earthmover.io/blog/cloud-native-dataloader

[josh28x]

Interested!