RuntimeError: CUDA error: out of memory when training the model for too long.

[safranchik]

There appears to be a memory leak during the validation routine since the GPU I'm using (A40, 48 GB VRAM) runs out of memory from training on the predict flu task for 125 epochs.

[TheMikeMerrill]

We discussed this on slack, but just wanted to confirm that this is a known bug. I believe there's a memory leak somewhere in on_train_epoch_end but I haven't been able to find it.

[TheMikeMerrill]

This isn't a complete solution, but I think the memory leak is happening in the metric bootstrapping during the validation loops. I can't figure out where specifically it's going down, but setting --model.val_bootstraps=0 after this commit (370d41f) should stop the leak.