- [2025/11] Omni-Effects is accepted by AAAI 2026 !
- [2025/08] We release the CogVideoX-1.5 finetuned on our Omni-VFX dataset !
- [2025/08] We release the controllable single-VFX/Multi-VFX version of Omni-Effects!
case1.mp4 |
case2.mp4 |
case3.mp4 |
case4.mp4 |
case5.mp4 |
case6.mp4 |
case7.mp4 |
case8.mp4 |
Visual effects (VFX) are essential visual enhancements fundamental to modern cinematic production. Although video generation models offer cost-efficient solutions for VFX production, current methods are constrained by per-effect LoRA training, which limits generation to single effects. This fundamental limitation impedes applications that require spatially controllable composite effects, i.e., the concurrent generation of multiple effects at designated locations. However, integrating diverse effects into a unified framework faces major challenges: interference from effect variations and spatial uncontrollability during multi-VFX joint training. To tackle these challenges, we propose Omni-Effects, a first unified framework capable of generating prompt-guided effects and spatially controllable composite effects. The core of our framework comprises two key innovations: (1) LoRA-based Mixture of Experts (LoRA-MoE), which employs a group of expert LoRAs, integrating diverse effects within a unified model while effectively mitigating cross-task interference. (2) Spatial-Aware Prompt (SAP) incorporates spatial mask information into the text token, enabling precise spatial control. Furthermore, we introduce an Independent-Information Flow (IIF) module integrated within the SAP, isolating the control signals corresponding to individual effects to prevent any unwanted blending. To facilitate this research, we construct a comprehensive VFX dataset Omni-VFX via a novel data collection pipeline combining image editing and First-Last Frame-to-Video (FLF2V) synthesis, and introduce a dedicated VFX evaluation framework for validating model performance. Extensive experiments demonstrate that Omni-Effects achieves precise spatial control and diverse effect generation, enabling users to specify both the category and location of desired effects.
git clone https://github.com/AMAP-ML/Omni-Effects.git
cd Omni-Effects
conda create -n OmniEffects python=3.10.14
pip install -r requirements.txtDownload checkpoints from HuggingFace and put them under checkpoints folder.
We have released the most comprehensive VFX dataset currently available on HuggingFace. The dataset primarily consists of three sources: assets from Open-VFX dataset, distillations of VFX provided by Remade-AI, and VFX videos created using FLF2V. Due to copyright restrictions, a small portion of the videos cannot be publicly shared. Additionally, we provide the CogVideoX1.5-5B-I2V-OmniVFX, fine-tuned on our Omni-VFX dataset. This model enables prompt-guided VFX video generation. The current supported prompts are in VFX-prompts.txt.
sh scripts/prompt_guided_VFX.sh # modify the prompt and input imageCurrent SPA-guided spatially controllable VFX supports controllable "Melt it", "Levitate it", "Explode it", "Turn it into anime style" and "Change the setting to a winter scene". We provide the corresponding LoRA based on CogVideoX-5B-I2V.
sh scripts/inference_omnieffects_singleVFX.shsh scripts/inference_omnieffects_multiVFX.shOmni-Effects achieves precise spatial control in visual effects generation.
We would like to thank the authors of CogVideoX, EasyControl, Switch_Transformers and VFXCreator for their outstanding work.
@article{mao2025omni,
title={Omni-Effects: Unified and Spatially-Controllable Visual Effects Generation},
author={Mao, Fangyuan and Hao, Aiming and Chen, Jintao and Liu, Dongxia and Feng, Xiaokun and Zhu, Jiashu and Wu, Meiqi and Chen, Chubin and Wu, Jiahong and Chu, Xiangxiang},
journal={arXiv preprint arXiv:2508.07981},
year={2025}
}