Long pauses during training when loading images from big `.mrcs`

[ff98li]

Hi,

I have been running CryoAI with this dataset EMPIAR-10076. However, for this particular dataset, because it stuffed all images into one big .mrcs file, CryoAI pause significantly long time during training.
Specifically, after every train_num_workers steps, the training gets stuck for about 5-10 minutes and GPU usage drops to 0%. I have compared the training time of EMPIAR-10028, where image chunks are split into multiple .mrcs, and that dataset trained smoothly without pausing.

Is there anyway to fix this?

Thanks!