Skip to content

minhtcai/unsloth-multi-gpu-vision

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 

Repository files navigation

unsloth-multi-gpu-vision

This is an extension from cwpeng's, you can now train Qwen3-VL with Unsloth on multi-GPU:

  1. For the first run, you need to disable multi-gpu for Unsloth to compile into unsloth_compiled_cache
  2. After that, add os.environ["UNSLOTH_COMPILE_DISABLE"] = "1", disable Unsloth compilation to avoid hanging <= I don't know why, this will reduce speed but my experiment is too small for noticeable effect
  3. The root cause seems to be related to gradient checkpointing
  4. Can run with DeepSpeed.
  5. Tested of 2x L4
  6. Working on GRPO for VL model.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages