I have tried to train this model on one GPU, but Tensorflow growth run out of system memory(32G) while 2k+ step training. I'm not sure it's because my modification(not much) or memory leaks or this code just consuming large memory.
I have tried to train this model on one GPU, but Tensorflow growth run out of system memory(32G) while 2k+ step training.
I'm not sure it's because my modification(not much) or memory leaks or this code just consuming large memory.