There is a high demand for audio-visual editing in video post-production and the film making field. While numerous models have explored audio and video editing, they struggle with object-level audio-visual operations. Specifically, object-level audio-visual editing requires the ability to perform object addition, replacement, and removal across both audio and visual modalities, while preserving the structural information of the source instances during the editing process. In this paper, we present \textbf{Object-AVEdit}, achieving the object-level audio-visual editing based on the inversion-regeneration paradigm. To achieve the object-level controllability during editing, we develop a word-to-sounding-object well-aligned audio generation model, bridging the gap in object-controllability between audio and current video generation models. Meanwhile, to achieve the better structural information preservation and object-level editing effect, we propose an inversion-regeneration holistically-optimized editing algorithm, ensuring both information retention during the inversion and better regeneration effect. Extensive experiments demonstrate that our editing model achieved advanced results in both audio-video object-level editing tasks with fine audio-visual semantic alignment. In addition, our developed audio generation model also achieved advanced performance. Demo--more video samples in our project page!
cd Object-AVEdit
# setup base environment
conda env create -f environment.yml
conda activate avedit
# install flash-attn3
git clone https://github.com/Dao-AILab/flash-attention
cd flash-attention/hopper
python setup.py install
# if run training
pip install colossalai --no-depsAudio generation weight.
Download from here and place it in \audio_weight.
Video generation weight.
Download from here and place it in \video_weight.
Other weights needed.
T5-large text encoder from here.
AudioLDM Mel vocoder from here.
We provide three types of tasks:
- Audio Generation Task.
- Audio Editing Task.
- Video Editing Task.
Fisrt, change the path in /audio_generation_model/generation.py
cd audio_generation_model
python generation.py --prompt "dog bark" --GPU_num=0 Fisrt, change the path in /audio_edit_part/audio_edit_main.py
cd /audio_edit_part
python audio_edit_main.py \
--input_audio "/demo_data/replace_data/21.wav" \
--source_prompt="Several brown cows are standing in a green alpine meadow under the tall, snowy mountains." \
--target_prompt="Several brown horses are standing in a green alpine meadow under the tall, snowy mountains." \
--edit_type="Replacement" \
--GPU_num=0 \
--word="cows" \
--seed=42 \
--CA=50 \
--SA=50 \
--threshold=0.1--word parameter means the target object that you want to edit. In replacement and removal tasks, the --word should be set to the source object to be edtied, and in addition task, --word shouldnot be set.
--threshold affects the mask process. Biger threshold means less change.
--CA means the cross attention control steps.
--SA means the cross attention control steps.
Biger CA or SA means more control steps. In removal task, SA is often set to 0.
Fisrt, change the path in /video_edit_part/config.py.
cd /video_edit_part
python video_edit_main.py \
--input_video "/demo_data/replace/6.mp4" \
--source_prompt="Several brown cows are standing in a green alpine meadow under the tall, snowy mountains." \
--target_prompt="Several brown horses are standing in a green alpine meadow under the tall, snowy mountains." \
--edit_type="Replacement" \
--GPU_num=0 \
--word="cows" \
--seed=42 \
--CA=37\
--SA=37 \
--threshold=0.05 --word parameter means the target object that you want to edit. In replacement and removal tasks, the --word should be set to the source object to be edtied, and in addition task, --word shouldnot be set.
--threshold affects the mask process. Biger threshold means less change.
--CA means the cross attention control steps.
--SA means the cross attention control steps.
Biger CA or SA means more control steps. In removal task, SA is often set to 0.
Our project is partly based on the Mochi-1, AudioLDM model. We would like to thank the authors for their excellent work! ❤️