A way to reduce bandwidth requirement for xDiT.

[Leechy-Litchi]

I found a project named CompactFusion，and it says

"CompactFusion transmits only the compressed residuals — the real information change — to drastically reduce bandwidth with minimal quality loss."

I hope this will resolve the high bandwidth requirement problem when most consumer-level GPU cannot use pcie p2p or nvlink directly.

[komikndr]

Oh wow, thanks! Looking at the code, I think it could be a drop-in replacement for Ring Attention. I also noticed a warmup pipeline, which is a signature feature of PipeFusion parallelism, something I’m currently working on that also significantly reduces communication bottlenecks. Thanks again for pointing this out. I really hope it’s a easy drop-in replacement.

[wallentri88]

@komikndr any new info here? Last time I tested Qwen Image bf16 and it worked 1.5x-2x slower in 2x 3090 setup than native 3090 + RAM offload. I suspect pci-e bandwidth of second card was the culprit...

[komikndr]

I think, for now, i bite more than i can chew about Pipefusion. I will try to implement cache or BNB NF4 first. Since the reason why i choose Pipefusion actually not because of reduce bandwidth, but more so about TP split. Pipefusion has those feature

Raylight works on FSDP to split model and not TP. However GGUF model is not really a native Torch dtype and distributed checkpoint loader, the one who responsible to spread the shard of FSDP, cannot accept weird GGUF tensor shape. Only TP can be done in GGUF that i know of.

And from the looks of its BNB store its data in native torch. I am sorry for not delivering my promise for now

About speed did you check if both of your GPU can p2p? nvidia-smi topo -p2p w

[wallentri88]

I see, so that's not as easy as it seemed. And it seems my cards/motherboard doesn't support p2p: all of them has CNS or NS statuses... Is p2p is a required feature to get any speedups?

[tillski791234]

If you have a mainboard with 8x8x bifurcation switch to 2 16x ports or you have a riser and your board supports 8x8x bifurcation on the 16x port, this might work (min. Gen4!). (24 lanes are a joke of the chipmanufacturer)
At best a Threaddripper-Combo or Intel Xeon similar with enough PCIe-lanes. But who does this?
With 13GB/s I´m at the end of 8x Gen4 and everything is bottlenecking. No speed advantage for me - bandwidth stop-and-forwards are eating up all speed. A well tuned KSampler is always faster on one GPU on upperend 40xx or 50xx (for now, but komikndr seems to work on this - and raylight is veryvery new - and the first real working GPU-multithreading approach for comfy. I think we have to be patient.)

NS: Look here for a p2p driver patch for consumer RTX: (maybe you have)
https://github.com/aikitoria/open-gpu-kernel-modules

And here for p2p debugging (helped me well)
https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5

Your GPUs have to be direct on CPU lanes, there is no other way yet I think. NVLink is rare.

[Leechy-Litchi]

well, I had tested some models in my dual V100 server(with 300G nvlink), it can be about 130%~170% boost, but if I switched to pcie, it will be even half the speed of single V100（I think it was influenced by pcie3.0, but i have no other hardware to compare）. Unfortunately, the price of ram is increasing so fast that i cannot add more ram, i could only use lots of swap to use ray nodes, so i didn't make enough tests. I hope this will help others for predicting the speed between nvlink and just pcie（without p2p）, and i will try to run with same args in readme to compare.(uh, maybe)
（I use NCCL_P2P_LEVEL=XXX to switch, and it seems that V100 cannot use pciep2p directly, and my test environment is a VM which someone mentioned it may cause compatibility problem, so it will take sometime for me to test the bottleneck）

followings are my test results using fsdp（only test once, so it won't be really reliabe）
flux.1 fp8e5m2 quant
1024x1536 batch=4 20steps
V100 168s
dual V100 96s
（In this case, single V100 seems OOM so it looks much slower）

512x512 batch=8 20steps
V100 46s
dual V100 32s

1920x1088 batch=2 20steps（it's not a typo, i forgot to replace the SD latent node）
V100 125s
dual V100 71s
（so, compare to 512x512x8，larger input seems better？）

Unfortunately, I found qwen-image doesn't have official fp16 model, so the precision is upcast to fp32, and yes, it cannot use tensorcore in this case too, GPU after ampere may be much faster with bf16 support, or use this model （it's too huge, i cann't use it with only two 16G V100, if z-image have one, i will test it）