the reconstruction is collapsing at the head rotation

[allamejust]

Dear authors,
thanks for the amazing work.

Based on your code, I am working on implementing an online demo for real-time avatar generation. Currently, the reconstruction produces a high-quality frontal face, but as soon as the head starts to rotate, the reconstruction quality begins to degrade significantly, as shown in the attached figure.
frontal face frame

rotated face frame

I am not entirely sure of the exact reason for this degradation. However, I noticed in your demo that you let the Gaussian model train on head rotations for a while before starting the reconstruction process. Could you please elaborate on this stage?
https://gapszju.github.io/RGBAvatar/live_demo_s.mp4
Additionally, I am curious why this stage was not included in either your train_offline.py or train_online.py files. Is it an intentional omission, or was it handled in a different part of your workflow?

Thanks,

[LinzhouLi]

Hi,

Thank you for your interest in our work and for sharing your observations.

From the figure you attached, it seems that even the frontal-view reconstruction quality is not very high — this usually happens in the very early stage of training. In online training, when the avatar’s head turns to the side, the immediate rendering quality will indeed drop because the model has not yet seen side-view data. However, it should quickly adapt and fit to the new views, as demonstrated in our demo video.

In our online reconstruction demo, the subject does rotate their head before making other expressions. This is simply a natural way to collect more diverse training views (the subjects in the INSTA dataset also tend to do this). Our algorithm does not treat this as a special stage — in train_online.py, the data is fed sequentially in the order it arrives, without splitting it into multiple stages.

Best regards,
Linzhou Li