-
Notifications
You must be signed in to change notification settings - Fork 6
LoadFileMixin always opens the whole file #459
Copy link
Copy link
Open
Labels
Description
Description
The LoadFileMixin is used by rasterio, csv, and xarray nodes for both local and remote files. As written, the it always loads the whole file. The Mixin should be updated to only load the whole file in certain cases, and otherwise open the dataset directly from the source path.
Describe the solution you'd like
Distinguish between local and remote sources. Only read the file into memory if necessary for caching.
Additional Notes
The Rasterio node should be modified to not use the mixin. This will make the read_from_source unnecessary.
Reactions are currently unavailable