Question: How did you mitigate ERA5 I/O bottlenecks / huge dataset reads?

[liname3]

Hi, authors!

I’m trying to fine-tune your excllent aurora model to my downstream tasks and I’m running into a serious I/O bottleneck with ERA5 due to the dataset size—data loading dominates and GPU utilization stays low. At the moment I store each training sample as HDF5 file, but I suspect the per-file overhead and random reads are killing throughput. Could you share how you handled the ERA5 “huge data + I/O bottleneck” problem in your pipeline?

If you have scripts/configs or pointers in the repo for preprocessing and recommended layout, I’d really appreciate them—thanks!

Thank you again for your work and your time.

[wesselb]

Hey @liname3! Thanks for opening an issue. :)

Data loading is indeed challenging. We unfortunately have no scripts or pointers in the repo that we can share. The general approach to efficient data loading is to store everything that you need for one batch in a separate file. This way you don't load more data than you use. E.g., if you store all of 2T for the entire month in a file, then you would need to read that whole file just to get a single hour, which is not efficient. Once you've stored batches in separate files, you can parallelise reading these files with PyTorch's DataLoader and num_workers=3, for example. This should give acceptable data loading performance.