Dreambooth + AnimateDiff + ControlNet

[tumurzakov]

As I understand you making following steps:

1. Fine tune dreambooth on all frames
2. Fine tune animatediff motion module
3. Extract controlnet from video
4. Combine them to infer

Can you share hyperparameters of finetuning?

1. Do you finetune LoRA or Dreambooth? How many steps, what lr?
2. Do you finetune all module or selected layers?

[liewjunhao]

Hi, we do not finetune DreamBooth/LoRA on the input video frames. We simply replace the Stable Diffusion with the personalized model downloaded from CivitAi for different styles.

[tumurzakov]

After you published your paper, I conducted experiments involving fine-tuning a U-Net in conjunction with a motion module, and it yielded promising results for style transfer. Prior to this, I explored LoRA, but the outcomes were less than impressive. I divide the samples into segments of video_length (16 frames) and then train the U-Net with the motion module on these segments as a single batch.

I have observed that the motion module, equipped with a single transformer block, possesses enough capability to capture simple motions. However, it does face limitations in memorizing shapes due to the absence of convolutional layers. Fine-tuning the U-Net, on the other hand, empowers it to retain both textures and shapes.

[wdapp]

你好，我想问一下 MagicEdit 预计什么时候能上线使用呢

[lmx99]

MagicEdit 哪个时候可以使用到呢？