How does your method compare with SyncTalk on long template video inputs?

[kudoNCT]

Hi authors, thanks for releasing the code! How does your method compare with SyncTalk on long template video inputs ( > 60 seconds) ?

[Fictionarry]

Hi, thanks for the interest! We had a test between InsTaG, TalkingGaussian, and ER-NeRF using the full training data of the 4 English videos when applying DeepSpeech and found InsTaG still performs better, as shown in Table 11 in the paper. Additionally, we tested SyncTalk under the same setting (using the same head poses and without the merging of gt head background), and found it performs slightly worse than TalkingGaussian. I think the conclusion will also hold true under other fair conditions, such as using HuBERT or AVE, more accurate head poses, or merging the synthesized face and GT head like SyncTalk proposed.

[kudoNCT]

Thank you for your response. Your explanation is very detailed, and it helps me a lot.

[kudoNCT]

Hello @Fictionarry , I’ve noticed that when using the input transforms.json from face_tracker.py compared to using the input transforms.json from Synctalk’s solution, your method produces better results for some video templates, but in certain cases, Synctalk yields better results. Could you explain the differences between the two methods for generating transforms.json, and when should one use each method?

[Fictionarry]

Hello @Fictionarry , I’ve noticed that when using the input transforms.json from face_tracker.py compared to using the input transforms.json from Synctalk’s solution, your method produces better results for some video templates, but in certain cases, Synctalk yields better results. Could you explain the differences between the two methods for generating transforms.json, and when should one use each method?

Hi, I haven't deeply dug into SyncTalk's implementation. But according to their paper, the difference is that they additionally add an optical flow bundle adjustment to help reduce the errors in pose estimation (usually causing head jittering) in the original face_tracker.py that is simply based on 3DMM fitting, which is used in RAD-NeRF, ER-NeRF, TalkingGaussian, and this work and etc.

I personally prefer the original one because of its simplicity and some of my personal concerns about the robustness of the optical flow model. And for the research purpose, I am not really sensitive to the head jittering. And when using the pose from the original one, adding some smoothness like this during inference can help to reduce the jittering of the head as well. It may be hard to say which one is better because I do not have much experience with the one of SyncTalk yet. You can arbitrarily choose one you like.

[kudoNCT]

Thank you for your response. Can I use smoothing during training, or is it only for inference?

[Fictionarry]

Thank you for your response. Can I use smoothing during training, or is it only for inference?

I think it's better only used in inference. This is mainly for improving visualization but not the metrics.