CUDA_ERROR_OUT_OF_MEMORY: out of memory / on MSN-Hard training

[alexcbb]

Hello, thank you for reproducing the work of the paper.

I tried to launch the training of the model on the MSN-Hard dataset, but I'm unable to launch the training because a CUDA_ERROR_OUT_OF_MEMORY error that I get at the beginning of the training.

The training is launched on a cluster and more specifically on a node containing 8 * GPU V100 (32Go each)
I further reduced the batch size to 32 and the sampled points to 2300, but it do not seems to suffice to avoid the error.

Here is the command that I use to launch the code (inside a .slurm file containing the necessary parameters) :
torchrun --standalone --nnodes 1 --nproc_per_node 8 train.py runs/msn/osrt/config.yaml

I have to point out that I was able to make the model train on the CLEVR3D dataset successfully following the same procedure.

To load the dataset, I downloaded the following : https://console.cloud.google.com/storage/browser/kubric-public/tfds/kubric_frames/multi_shapenet_conditional, putting the 2.8.0/ folder inside data/osrt/multi_shapenet_frames/ folder of the project.

Did you ever encounter this issue ?

Thank you

[alexcbb]

It seems that the issue is linked with Tensorflow, I'm still investigating

[takerum]

Hi, I encountered the same problem, and a workaround that works for me is to tell TensorFlow not to use GPUs explicitly just after this line https://github.com/stelzner/osrt/blob/main/osrt/data/multishapenet.py#L24:

import tensorflow as tf
tf.config.set_visible_devices([], 'GPU')

or you could uninstall the current TensorFlow and re-install cpu version of Tensorlow (tensorflow-cpu).

[alexcbb]

Hi, I encountered the same problem, and a workaround that works for me is to tell TensorFlow not to use GPUs explicitly just after this line https://github.com/stelzner/osrt/blob/main/osrt/data/multishapenet.py#L24:

import tensorflow as tf
tf.config.set_visible_devices([], 'GPU')

or you could uninstall the current TensorFlow and re-install cpu version of Tensorlow (tensorflow-cpu).

It looks to have resolved my problem, I close the issue now, thank you !

[alexcbb]

Finally after 10 hours, the training seems to never begin. I don't have any errors but only warnings

[alexcbb]

It seems that the problem comes from the part in the MSN dataset where it should iterate through the data in numpy :

(in the __iter__ of the multishapenet.py file)
tf_iterator = dataset.as_numpy_iterator()

for data in tf_iterator:
    yield self.prep_item(data)

I put logs inside this part of the code and find out that it transforms the dataset as a numpy iterator succesfully but the script is stuck in the for loop : it does not enter it and is not able to iterate over the data.

[stelzner]

Hi, thank you both for raising the issue and finding a potential fix. I didn't run into this as I had tensorflow-cpu installed.

@alexcbb Could you confirm that your Tensorflow installation works, by running some simple test operations on both CPU and GPU? I've seen it silently fail like that when there are e.g. driver issues. Maybe you could also try tensorflow-cpu to see if that changes things.

[alexcbb]

@stelzner
I tried with both solutions : either using the tensorflow-gpu with the given workaround, either using tensorflow-cpu, but the problem is still identical: i'm blocked at the point indicated in my previous message.

Normally the drivers should be correct because I load modules pre-created for the cluster I'm using. I will try some tests on the tensorflow installation but it should'nt be the cause of the problem (shouldn't it only be using the CPU anyway ?) By the way, would it be linked to the version of tensorflow itself ? Which tensorflow-gpu are you using ?

Thank you very much for your time !

[alexcbb]

It seems that setting 'num_workers' to 0 and deactivating the 'persistent_workers' option for the train_dataloader stops the blocking for me (maybe it's something linked to the way the cluster I use is parametrized).

Can you explain the use of those parameters further @stelzner, are they really necessary for the training ? (At first I didn't want to change it because of the assert in train.py)

[stelzner]

Interesting, glad you found a solution! My idea was that Tensorflow has its own parallelization scheme, so invoking it multiple times from the PyTorch side seemed redundant. Also, I wanted to avoid reinitializing it after each epoch, hence I set persistent_workers=True. So training should work correctly after modifying these parameters, I would have just expected it to be slower. But clearly there's something unexpected going on here!

[chenguolin]

It works for me by

1. setting num_workers to 0;
2. adding the following scripts after import sunds in https://github.com/stelzner/osrt/blob/main/osrt/data/multishapenet.py#L24:

import tensorflow as tf
tf.config.set_visible_devices([], "GPU")  # disable GPU usage for Tensorflow
for device in tf.config.get_visible_devices():
    assert device.device_type != "GPU"

I have to say it's a nightmare to use TensorFlow 😰