torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 400.00 GiB

[Go1denMelody]

When I run video super resolution model, there is an error
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 400.00 GiB (GPU 0; 44.52 GiB total capacity; 12.21 GiB already allocated; 31.33 GiB free; 12.83 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Why it need to try to allocate 400gb, should i change some settings?

[Go1denMelody]

Traceback (most recent call last):
File "/home/powerop/work/LaVie/vsr/sample.py", line 151, in
main(OmegaConf.load(args.config))
File "/home/powerop/work/LaVie/vsr/sample.py", line 109, in main
upscaled_video_ = pipeline(
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/powerop/work/LaVie/vsr/models/pipeline_stable_diffusion_upscale_video_3d.py", line 766, in call
image = self.decode_latents_vsr(latents[start_f:end_f])
File "/home/powerop/work/LaVie/vsr/models/pipeline_stable_diffusion_upscale_video_3d.py", line 356, in decode_latents_vsr
image = self.vae.decode(latents).sample
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/diffusers/utils/accelerate_utils.py", line 46, in wrapper
return method(self, *args, **kwargs)
File "/home/powerop/work/LaVie/vsr/models/autoencoder_kl.py", line 197, in decode
decoded = self._decode(z).sample
File "/home/powerop/work/LaVie/vsr/models/autoencoder_kl.py", line 184, in _decode
dec = self.decoder(z)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/diffusers/models/vae.py", line 233, in forward
sample = self.mid_block(sample)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/diffusers/models/unet_2d_blocks.py", line 463, in forward
hidden_states = attn(hidden_states)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/powerop/work/LaVie/myenv/lib/python3.10/site-packages/diffusers/models/attention.py", line 162, in forward
hidden_states = F.scaled_dot_product_attention(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 400.00 GiB (GPU 0; 44.52 GiB total capacity; 12.21 GiB already allocated; 31.35 GiB free; 12.80 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[Go1denMelody]

After checking, I found there is something wrong about xformers when I install requirements. After fixing it, the code can run successfully

[johndpope]

close

[zillur01]

Hi, what was the issue you found on xformer? I am getting the same error