Colab dreambooth notebook fail

[andrewssdd]

Describe the bug

The Dreambooth Colab notebook fails at the training stage. Seems to be an issue with bitsandbytes.

Reproduction

Run the Dreambooth Colab notebook. It fails at training.

https://colab.research.google.com/github/ShivamShrirao/diffusers/blob/main/examples/dreambooth/DreamBooth_Stable_Diffusion.ipynb

Exception: CUDA SETUP: Setup Failed!

Logs

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
For effortless bug reporting copy-paste your error into this form: https://docs.google.com/forms/d/e/1FAIpQLScPB8emS3Thkp66nvqwmjTEgxp8Y9ufuWTzFyr9kJ5AoI47dQ/viewform?usp=sf_link
================================================================================
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:105: UserWarning: /usr/lib64-nvidia did not contain libcudart.so as expected! Searching further paths...
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/sys/fs/cgroup/memory.events /var/colab/cgroup/jupyter-children/memory.events')}
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('8013'), PosixPath('//172.28.0.1'), PosixPath('http')}
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//colab.research.google.com/tun/m/cc48301118ce562b961b3c22d803539adc1e0c19/gpu-t4-s-3a2kk3hhilbsk --tunnel_background_save_delay=10s --tunnel_periodic_background_save_frequency=30m0s --enable_output_coalescing=true --output_coalescing_required=true'), PosixPath('--logtostderr --listen_host=172.28.0.12 --target_host=172.28.0.12 --tunnel_background_save_url=https')}
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/datalab/web/pyright/typeshed-fallback/stdlib,/usr/local/lib/python3.10/dist-packages')}
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('/env/python')}
  warn(
/usr/local/lib/python3.10/dist-packages/bitsandbytes/cuda_setup/paths.py:27: UserWarning: WARNING: The following directories listed in your path were found to be non-existent: {PosixPath('//ipykernel.pylab.backend_inline'), PosixPath('module')}
  warn(
CUDA_SETUP: WARNING! libcudart.so not found in any environmental path. Searching /usr/local/cuda/lib64...
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so
CUDA SETUP: Highest compute capability among GPUs detected: 7.5
CUDA SETUP: Detected CUDA version 122
CUDA SETUP: TODO: compile library for specific version: libbitsandbytes_cuda122.so
CUDA SETUP: Defaulting to libbitsandbytes.so...
CUDA SETUP: CUDA detection failed. Either CUDA driver not installed, CUDA not installed, or you have multiple conflicting CUDA libraries!
CUDA SETUP: If you compiled from source, try again with `make CUDA_VERSION=DETECTED_CUDA_VERSION` for example, `make CUDA_VERSION=113`.
Traceback (most recent call last):
  File "/content/train_dreambooth.py", line 869, in <module>
    main(args)
  File "/content/train_dreambooth.py", line 571, in main
    import bitsandbytes as bnb
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/__init__.py", line 6, in <module>
    from .autograd._functions import (
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py", line 5, in <module>
    import bitsandbytes.functional as F
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/functional.py", line 13, in <module>
    from .cextension import COMPILED_WITH_CUDA, lib
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py", line 41, in <module>
    lib = CUDALibrary_Singleton.get_instance().lib
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py", line 37, in get_instance
    cls._instance.initialize()
  File "/usr/local/lib/python3.10/dist-packages/bitsandbytes/cextension.py", line 27, in initialize
    raise Exception('CUDA SETUP: Setup Failed!')
Exception: CUDA SETUP: Setup Failed!

System Info

Google Colab

[andrewssdd]

Here's the requirement cell that works.

!wget -q https://github.com/ShivamShrirao/diffusers/raw/main/examples/dreambooth/train_dreambooth.py
!wget -q https://github.com/ShivamShrirao/diffusers/raw/main/scripts/convert_diffusers_to_original_stable_diffusion.py
%pip install  git+https://github.com/ShivamShrirao/diffusers
%pip install  -U --pre triton
%pip install  transformers ftfy bitsandbytes gradio natsort safetensors xformers torch==2.1.0+cu121 accelerate

[VladAdushev]

Here's the requirement cell that works.

returns an error

[TragicXxBoNeSxX]

Still returns "Exception: CUDA SETUP: Setup Failed!"

[vadar007]

Was able to successfully execute with modified code. However, when attempting to use the generated model with Stable Diffusion was getting the following error:

*** Error verifying pickled file from D:\l......*.ckpt
*** The file may be malicious, so the program is not going to read it.
*** You can skip this check with --disable-safe-unpickle commandline argument.

Adding recommended command line argument allowed Stable Diffusion to utilize the model

[TragicXxBoNeSxX]

Changing this line: %pip install transformers ftfy bitsandbytes gradio natsort safetensors xformers torch==2.1.0+cu121 accelerate

To this: %pip install transformers ftfy bitsandbytes gradio natsort safetensors xformers torch==2.1.0+cu121 accelerate kaleido cohere openai tiktoken

Got it working for me again.

[andrewssdd]

Was able to successfully execute with modified code. However, when attempting to use the generated model with Stable Diffusion was getting the following error:

*** Error verifying pickled file from D:\l......*.ckpt *** The file may be malicious, so the program is not going to read it. *** You can skip this check with --disable-safe-unpickle commandline argument.

Adding recommended command line argument allowed Stable Diffusion to utilize the model

You need to save the checkpoint as safetensors

[Al-Rien]

To this: %pip install transformers ftfy bitsandbytes gradio natsort safetensors xformers torch==2.1.0+cu121 accelerate kaleido cohere openai tiktoken

Still returns CUDA SETUP: Setup Failed.

[TianyiPeng]

Does anyone get it to work now?

[domingosl]

Same issue here after trying all suggestions, CUDA SETUP: Setup Failed

[abc123desygn]

I have the same issue. Can you please fix?

[tibor]

I’m not sure but I think this is a problem with the library bitsandbytes. I have opened a ticket here: bitsandbytes-foundation/bitsandbytes#950

[VladAdushev]

Has anyone managed to launch it?

[deveshruttala]

same issue with me

[tibor]

Yeah, it works with the fix at bitsandbytes-foundation/bitsandbytes#950