About the selection of input video pairs #15

[TimZhang001]

I have an idea. For the same prompt, can I generate videos on wan1.3B and wan14B respectively, and then use the video generated by 14B to DPO optimize the 1.3B one? I don't know if it is feasible. Or when doing DPO, must the video generated by the current model be used?

[CIntellifusion]

That's valuable thought. We have tried to use vc2 dataset to finetune vc1 checkpiont which doesn't work since the dataset between the two versions of videocrafter is quite different. If the p_theta_wan14B and p_theta_wan1.3B is similar, I think it will work.
Theoretically, the training data and the init model checkpoint should from same disctribution, noted as p_ref in paper.
If you are interested, you could give it a quick try! I am excited to discuss it further if there's experiment result.
If wan14B data can't work for WAN1.3B dpo, there is an alternative experiment setting that could possibly work:

Train a large model and a tiny model on same data, and get a large model ckpt that performs better than tiny model. Then use data genereted by large model to DPO finetune tiny model.

I think this idea is also have some connection with data distillation like DMD/DMD2.

My personal email is 2928826539@qq.com and wuhaoyu556@ruc.edu.cn for further discussion or cooperation.

[suruoxi]

Hallo4 use different models (even GANs) to generate videos. But it limits the video varitants by leveraging ref image and pose control.