Possible to get native support as part of ComfyUI Core?

[doogyhatts]

Just wondering whether if Ruyi can be supported as part of ComfyUI core, just like Hunyuan.
This will open up GGUF support from city96.
Otherwise, you might have to do your own quantisation, or use a different quantisation method such as SVDQuant.
This will allow people with lower-tier GPUs to use Ruyi.

[cellzero]

That sounds good. I will look into how Hunyuan is integrated with ComfyUI.

I think it could be possible with some modifications.

[doogyhatts]

Today's blog on Comfy briefly describes their VAE decoding node having a temporal tiling solution that reduces overall vram usage for Hunyuan video to 8gb for fp8 precision. This should be another way to reduce the overall vram usage for Ruyi, and ensures that Ruyi becomes compatible to use Comfy's optimisation features.

[clover1980]

I support the need for improving speed generation. On RTX 4070 Ti Super any generation takes 40 minutes and it's just for 5 sec gif output.
Very strange thing that changing schedulers (in your original Comfy script) are not affecting speed at all, Euler which is simpler and can make worse quality than DDIM generates exactly in same 40 minutes on same GPU. Why there's even such choice if it affects only quality?
And of course the longer length of video need to be provided, i can't change generation more than 5 seconds output at all. I would even wait longer for longer video.

[doogyhatts]

As for improving the speed generation, I would suggest adding flash/sage attention as well.

[cellzero]

I have tested FlashAttention and SageAttention locally.

FlashAttention performs slightly better than the PyTorch version of SDPA Flash Attention.

As for SageAttention, since the head_dim of Ruyi is larger than what SageAttention supports (176 vs 128), Ruyi cannot benefit from SageAttention. I attempted to modify SageAttention's code to support 176 head_dims, but this resulted in increased generation time and higher GPU memory usage. Sadly.