questions about the up axis

[xiaoxiongweimi]

Sorry, I have a question. Why the up axis of smpl skeleton is z up. I print out the offsets of smpl skeleton, it should be y up. Is there something you considered and I missed that information?

[c-he]

Hi,
Though the skeleton is y up, the AMASS dataset we use targets z up. If you use its Blender addon, you can see they have a special rotation applied to adjust the orientation for AMASS data (https://gitlab.tuebingen.mpg.de/jtesch/smplx_blender_addon/-/blob/master/__init__.py#L1116).

[xiaoxiongweimi]

Thank your for your reply. If I only want to train the generative model (not including the gmp model), maybe I should set the 'output_trans' as 'True' and the corresponding 'global_output' as 18, right?

Hi, Though the skeleton is y up, the AMASS dataset we use targets z up. If you use its Blender addon, you can see they have a special rotation applied to adjust the orientation for AMASS data (https://gitlab.tuebingen.mpg.de/jtesch/smplx_blender_addon/-/blob/master/__init__.py#L1116).

[c-he]

Yes, also make sure to leave pretrained_gmp blank.

[xiaoxiongweimi]

I trained the model on our dance data and can see a gradual reduce of losses except the trans loss. The curve is very abnormal. According to the scripts, the trans loss is calculated via output 'root velocity' and 'root height' rather than the direct output 'trans', so can this curve be a possible case?

Yes, also make sure to leave pretrained_gmp blank.

[c-he]

It looks a bit weird. You said you trained the model on your dance data, could you make sure these data are y-up or z-up?

[xiaoxiongweimi]

Our data are y-up, and I modified the config in the yaml file. I think there must be something wrong when I preprocessed our data. Sigh~~~

It looks a bit weird. You said you trained the model on your dance data, could you make sure these data are y-up or z-up?

[xiaoxiongweimi]

It looks a bit weird. You said you trained the model on your dance data, could you make sure these data are y-up or z-up?

I solved the problem about trans loss, it is about the parameter dimension when I call "estimate_angular_velocity" and "estimate_linear_velocity", and then could you tell me the difference between gmp datasets and generative datasets. It seems there are links between them but not exactly the same. I currently use the same datasets when I train gmp and generative, but the result goes too far away when gmp model predict the 'trans'.

[c-he]

Glad to know you solved your problem! Yes the processing steps for gmp and generative datasets are a bit different. When processing AMASS data for our generative setup, we used the configuration provided in https://github.com/c-he/NeMF/blob/main/configs/amass.yaml. While processing AMASS data to train our gmp, we set unified_orientation to False since we need the global orientation info to correctly predict global translations. Therefore, when combining these 2 trained models, you can see we insert the root orientation to obtain joint rotations and positions, and then compute other necessary gmp input data based on them.

NeMF/src/nemf/generative.py

Lines 233 to 251 in 146a1ea

	elif self.args.pretrained_gmp:
	local_rotmat = local_rotmat.view(b_size, -1, n_joints, 3, 3) # (B, T, J, 3, 3)
	if self.args.data.root_transform:
	root_orient_rotmat = rotation_6d_to_matrix(output['root_orient']) # (B, T, 3, 3)
	local_rotmat[:, :, 0] = root_orient_rotmat
	pos = torch.matmul(root_orient_rotmat.unsqueeze(2), pos_recon.unsqueeze(-1)).squeeze(-1)
	else:
	pos = pos_recon
	gmp_data = dict()
	gmp_data['rot6d'] = matrix_to_rotation_6d(local_rotmat) # (B, T, J, 6)
	gmp_data['angular'] = estimate_angular_velocity(local_rotmat.clone(), dt=dt) # (B, T, J, 3)
	gmp_data['pos'] = pos
	gmp_data['velocity'] = estimate_linear_velocity(pos, dt=dt)
	pred_data = self.gmp.predict(gmp_data, dt=dt)
	output |= pred_data # merge two dicts

[xiaoxiongweimi]

Glad to know you solved your problem! Yes the processing steps for gmp and generative datasets are a bit different. When processing AMASS data for our generative setup, we used the configuration provided in https://github.com/c-he/NeMF/blob/main/configs/amass.yaml. While processing AMASS data to train our gmp, we set unified_orientation to False since we need the global orientation info to correctly predict global translations. Therefore, when combining these 2 trained models, you can see we insert the root orientation to obtain joint rotations and positions, and then compute other necessary gmp input data based on them.

NeMF/src/nemf/generative.py

Lines 233 to 251 in 146a1ea

	elif self.args.pretrained_gmp:
	local_rotmat = local_rotmat.view(b_size, -1, n_joints, 3, 3) # (B, T, J, 3, 3)
	if self.args.data.root_transform:
	root_orient_rotmat = rotation_6d_to_matrix(output['root_orient']) # (B, T, 3, 3)
	local_rotmat[:, :, 0] = root_orient_rotmat
	pos = torch.matmul(root_orient_rotmat.unsqueeze(2), pos_recon.unsqueeze(-1)).squeeze(-1)
	else:
	pos = pos_recon
	gmp_data = dict()
	gmp_data['rot6d'] = matrix_to_rotation_6d(local_rotmat) # (B, T, J, 6)
	gmp_data['angular'] = estimate_angular_velocity(local_rotmat.clone(), dt=dt) # (B, T, J, 3)
	gmp_data['pos'] = pos
	gmp_data['velocity'] = estimate_linear_velocity(pos, dt=dt)
	pred_data = self.gmp.predict(gmp_data, dt=dt)
	output |= pred_data # merge two dicts

Thank you~ And when combining these two models, is it more reasonable to denormalize the generative output first and then compute gmp input based on denormalized data because when gmp predicting, it will do normalization just like when it is trained.

[c-he]

In our setup, we only normalize the input data and let our network output unnormalized results. Therefore, it's unnecessary to perform denormalization when combining these two models.