Check out Burla's 1 Trillion Row Challenge demo: https://docs.burla.dev/examples/process-2.4tb-in-parquet-files-in-76s
It seems like with this library as it is today, we could use Burla's remote_parallel_map and perform highly scalable, efficient distributed SQL processing on Xarray data.
I think this could help us get this project to the 100 Trillion row challenge (#34) in a really short amount of time.
CC: @JacobZuliani, @jayendra-info.