question about global orientation and translate

[Exusial]

Hi Shenhan, thank you for your great work! I'm working on refining flame model out of a SMPLX estimation using VHAP. However, I find the photometric loss is always inf, which means nothing is rendered with given camera parameters. I found that global orientations and translate will move the whole body (flame head included) drastically, which makes the flame model far away from its canonical position. My question is: should I first translate the flame model using the global orientation and translate parameters? Or should I change the way of initializing the flame parameters in tracker.py?

self.shape = torch.zeros(self.cfg.model.n_shape).to(self.device)
self.expr = torch.zeros(self.n_timesteps, self.cfg.model.n_expr).to(self.device)

self.neck_pose = torch.zeros(self.n_timesteps, 3).to(self.device)
self.jaw_pose = torch.zeros(self.n_timesteps, 3).to(self.device)
self.eyes_pose = torch.zeros(self.n_timesteps, 6).to(self.device)

self.translation = torch.zeros(self.n_timesteps, 3).to(self.device)
self.rotation = torch.zeros(self.n_timesteps, 3).to(self.device)

Thank you in advance for your help. Again, a fascinating job on VHAP!
Best regards.

[ShenhanQian]

Hi @Exusial,

photometric optimization is quite sensitive to initialization. The rendered FLAME model should be roughly aligned with the image before the optimization. Therefore, we disable rgb loss and only use landmark loss at the beginning to get a good initialization of the pose. You may need do a similar thing for your case.

You could also leave the saved visualization images here to share more details.

Cheers!

[Exusial]

Hi @ShenhanQian , Thank you for your advice! The landmark loss is calculated by projecting flame landmarks to 2d. Will it also be effected by the initialization? My case the predicted landmarks cannot align the the results of STAR, too. Should I add the global orientation first?

[ShenhanQian]

Hi @Exusial , there is a stage for VHAP that only optimizes the camera pose and global translation, and orientation with the landmark loss:

VHAP/vhap/config/base.py

Lines 228 to 231 in 4b64ab7

	class StageLmkInitRigidConfig(StageConfig):
	"""The stage for initializing the rigid parameters"""
	num_steps: int = 500
	optimizable_params: tuple[str, ...] = ("cam", "pose")

It is expected to provide a good initialization of the global pose. If STAR can't detect landmarks on the side views, you can also try face-alignment, which is less accurate in the front view but more robust to varying views.

[Exusial]

Hi @ShenhanQian, sorry for the late reply. In fact, the STAR can get a good initialization, but the lmkinitall stage fails to get aligned results:

[Fig. 1]
Will the results be affected by the global orientations?

[ShenhanQian]

Hi, how are the results of LmkInitRigid?

[Exusial]

Hi @ShenhanQian the green points are the results from LmkInit Rigid, and I guess the red points are the results of AllInit?

[Fig. 2]

Will camera parameters affect this stage?

[ShenhanQian]

The green points are the results of the landmark detectors, and the red points are the corresponding points on FLAME projected to 2D with your camera parameters.

According to Fig. 2, the landmark detection is good, but the FLAME model is not in the right place. This indicates either:

• The global translation and orientation are not properly optimized during LmkInitRigid stage
• The camera parameters are not properly used.

[Exusial]

@ShenhanQian Thank you for your reply! So it is not right that only green points appers in Fig. 2? I'll check my camera and also try to adopt prior translation before the init stage.

[ShenhanQian]

Yes, the red points should roughly align with the green ones at least after the LmkInitRigid stage.

[Exusial]

Thank you @ShenhanQian, I get VHAP work and export final image dataset. But the the training of GaussianAvatars cannot converge. I have two questions:

1. I have disabled camera relocation during the export of nerf dataset, will this effect the reconstruction process?
2. What is the camera model (OpenGL or OpenCV) of GaussianAvatars? I assume it is using the DynamicNerf loader for VHAP output?

[ShenhanQian]

hi, what do you mean by not converging?

1. It shouldn't hurt the performance
2. If you use the default arguments to run VHAP and GA, they should match. Yes, DynamicNerf loader

[Exusial]

Hi @ShenhanQian , I somewhat figure it out. Because i use online 3DGS visualizer (SuperSplat) to directly visualize the output point.ply. However that ply is binded to the flame mesh so that the GS should be first transformed to global space. But the results is still not good. I capture it as below.

GA.mp4

This is the training results of 60000 iter. I'd like to ask, whether this is because the training process has not converged, or is because of sparse input view (9 views), or is this due to inaccurate flame parameters predictions? Thank you.

[ShenhanQian]

Hi @Exusial , I would suggest using GA's viewer. It visualizes the process of optimization, making it a lot easier to analyze

[Vincento-Wang]

Hi Shenhan, thank you for your great work! I'm working on refining flame model out of a SMPLX estimation using VHAP. However, I find the photometric loss is always inf, which means nothing is rendered with given camera parameters. I found that global orientations and translate will move the whole body (flame head included) drastically, which makes the flame model far away from its canonical position. My question is: should I first translate the flame model using the global orientation and translate parameters? Or should I change the way of initializing the flame parameters in tracker.py?

self.shape = torch.zeros(self.cfg.model.n_shape).to(self.device)
self.expr = torch.zeros(self.n_timesteps, self.cfg.model.n_expr).to(self.device)

self.neck_pose = torch.zeros(self.n_timesteps, 3).to(self.device)
self.jaw_pose = torch.zeros(self.n_timesteps, 3).to(self.device)
self.eyes_pose = torch.zeros(self.n_timesteps, 6).to(self.device)

self.translation = torch.zeros(self.n_timesteps, 3).to(self.device)
self.rotation = torch.zeros(self.n_timesteps, 3).to(self.device)

Thank you in advance for your help. Again, a fascinating job on VHAP! Best regards.

same problem , wanna extend FLAME to SMPLX， have you achieved ?