Does xarray-sql load datasets lazily

[ahuang11]

Based on:
ctx.from_dataset('air', ds, chunks=dict(time=24)) # the dataset needs to be chunked!

It seems to be lazily loaded, but I wanted to confirm.

[alxmrs]

Hey Andrew, thanks, this is an excellent question.

Here is a more realistic demo colab notebook for xarray-sql: https://colab.research.google.com/drive/1TJ3I4I0UYbsUrxJ02vKm7UROJaJ8Za3V?usp=sharing.

In this notebook, loading a filtered down ERA5 monthly aggregate dataset takes ~50s to load. This suggests to me that xarray-sql is not currently lazy, despite our best efforts. This confirms what I suspected about the current way we register dataset with Arrow:

xarray-sql/xarray_sql/df.py

Line 65 in fa436e8

def from_map_batched(

Using a record batch reader seems to require a chunked scan over the dataset before creating the pyarrow Table.

We're currently working on a new way to register datasets into Datafusion via the Dataframe interchange protocol #86. According to my reading of the arrow docs and interchange protocol spec, this should provide a way for xarray-sql to load datasets in a truly lazy way.

[alxmrs]

I just read the sources from the pyarrow side of the dataframe interchange. Alas, I am skeptical if after this implementation, that loading xarray-sql will be lazy: https://arrow.apache.org/docs/_modules/pyarrow/interchange/from_dataframe.html#from_dataframe.

[alxmrs]

Update: @RohanDisa and I are exploring a new approach to achieve proper laziness (we've been pair programming for a couple of weeks).