Memory: multiple dataloaders with the same image dataset

[dyukha]

I have an image dataset, and, based on it, I create multiple dataloaders (I sample a different set of indices). At any point of time, only one dataloader is active. Basically, the code looks like this:

<create dataset writers>
for ....
    train_loader = Loader(....)
    <train and do other stuff using train_loader>

I'm confident that there are no leaks (the outcome of each iteration is a number, and I'm sure it's detached from the computational graph). However, I see my RAM usage growing linearly. Is there a way to clean up after the iterations? gc.collect() or torch.cuda.empty_cache() don't work.

I'm running this on google colab. Before, I was using the standard pytorch dataloaders, and there were no memory problems (I had cifar-10 dataset, and i wrapped it in the another dataset which sampled the indices I want).

Thank you!

[tavisshore]

What are your loader arguments?

[dyukha]

Loader(dataset_path,
                  batch_size=500,
                  num_workers=2,
                  order=OrderOption.SEQUENTIAL,
                  indices=indices, # The length can be up to 10^5, but the memory problems appear even for lengths of order 10^3
                  drop_last=False,
                  pipelines={'image': image_pipeline, 'label': label_pipeline})

I also tried os_cache=False, and I believe it made things much worse (to the point that even a single dataloader didn't finish).

Right now, I managed to make it work by creating a separate process for training with each train_loader (and it works 20x times faster than standard pytorch, so thank you very much for the library!). Still would be great to make it work without such hacks.