Mapping to relative joint positions / SMPL pose; skeleton in T pose

[kesdogan]

Hi @abcamiletto

I am Timur and a final year masters student at ETH Zurich. I discovered your work under the highlights of this year's CVPR and found it super impressive. The rapid speed your method achieves while being so accurate is insane.

I wanted to ask, whether you had given any thought about predicting relative pose parameters instead of absolute joint positions with your model? I am currently working on an avatar generation project, in which I would love to use your work to predict novel poses. However, it is SMPL based and thus takes SMPL pose parameter as an input.

Alternatively, do you have a T-pose version of the skeleton, that you could release, so that it would be possible to calculate the relative pose parameters analytically? I had checked out @jianwang-mpi 's skeleton implementation, that forms the basis for your paper as well, here. Unfortunately, his rest pose skeleton is also not released.

Really really love the work and thank you for publishing the code so quickly after CVPR.

Kind regards
Timur

[abcamiletto]

Hi Timur

Predicting relative pose parameters is possible, but in my experience, it performs a bit worse than predicting 3D keypoints (due to the error compounding down the skeleton). Relative pose parameters can always be recovered with an Inverse Kinematics algorithm with various success (something like https://github.com/zju3dv/EasyMocap might help).

The skeleton kinematic structure is in every sequence's meta folder, and I can paste it here too

[[null, 0], [0, 1], [1, 2], [2, 3], [3, 4], [4, 5], [5, 6], [6, 7], [7, 8], [4, 9], [9, 10], [10, 11], [11, 12], [12, 13], [13, 14], [14, 15], [15, 16], [12, 17], [17, 18], [18, 19], [19, 20], [12, 21], [21, 22], [22, 23], [23, 24], [12, 25], [25, 26], [26, 27], [27, 28], [12, 29], [29, 30], [30, 31], [31, 32], [12, 33], [4, 34], [34, 35], [35, 36], [36, 37], [37, 38], [38, 39], [39, 40], [40, 41], [37, 42], [42, 43], [43, 44], [44, 45], [37, 46], [46, 47], [47, 48], [48, 49], [37, 50], [50, 51], [51, 52], [52, 53], [37, 54], [54, 55], [55, 56], [56, 57], [37, 58], [0, 59], [59, 60], [60, 61], [61, 62], [62, 63], [0, 64], [64, 65], [65, 66], [66, 67], [67, 68]]

This is a JSON list with pairs of parent -> child pairs.

Keep in mind that this is the complete skeleton, and we only (currently) train with a subset of joints specified here.

In case you need more info than this, I cleaned up a minimal version of the skeleton in the form of a BVH, with a single motion frame zeroed out. This will give you the T pose you are looking for. It is saved as a .txt because GitHub does not allow .bvh attachments.

frame_skeleton_sample.txt

Hope that helps,
Good luck on your master thesis!

Andrea

[kesdogan]

Thank you so much for your answer @abcamiletto !

Did you try predicting relative pose parameters with this model? Is there code already to train a variant to predict relative pose parameters or would this need to be implemented anew?
Would love to open a pull request for that, in case I can contribute!

I have also thought about regular inverse kinematics. The issue is, that in order to get at SMPL pose parameters, the current 15 skeletal setup with 3D joint locations makes the problem under-constrained, so that it is hard to use the inverse kinematics methods of libraries such as pinocchio. For SMPL pose parameters, only a couple of additional 3D joint locations would need to be predicted [spine joints, pelivs (root), collar / shoulder joints].
Do you think predicting additional 3D joint locations is feasible? Or is there a good reason you decided to go with the 15 joint approach instead?

The rest pose skeleton you sent me contains 54 joints and is eerily similar to the SMPLX model, except for the placement of the collar joints [see fig, green circles]. Was this an intentional deviation on your part? Does the dataset you trained FRAME on contain all the 55 joint ground truth locations, so that it would be possible to train on more 3D joint locations?

Thank you so much for your help!
Timur

[abcamiletto]

Hi Timur,

Did you try predicting relative pose parameters with this model? Is there code already to train a variant to predict relative pose parameters or would this need to be implemented anew?

No code is available for that. But if you have the SMPL(X) parameters computed (via IK or whatever), that is an easy modification to make.
Happy to provide more details if you end up going for that direction.

I have also thought about regular inverse kinematics. The issue is, that in order to get at SMPL pose parameters, the current 15 skeletal setup with 3D joint locations makes the problem under-constrained, so that it is hard to use the inverse kinematics methods of libraries such as pinocchio. For SMPL pose parameters, only a couple of additional 3D joint locations would need to be predicted [spine joints, pelivs (root), collar / shoulder joints].

This is a common problem for which there is not clear-cut solution available IMO.

pinocchio is a library from the robotics world, but there are a few differences between how you do IK in robotics and with an SMPL model. In robotics, you usually have only the End Effector position (i.e. fingertips for humans), and you want to get whatever combination of joint angles to get there.

When working with SMPL, things are a bit different, as you do not only care about end effector but also about the joints' position along the kinematic chain. The problem is still under-constrained and you care a lot that your joints are in a reasonable range of values (i.e. the mesh around your elbow is not wrapped like a candy). The way this is addressed can vary a lot: in EasyMocap - if i remember correctly - they put some hard limits on which values the joints can be, in VPoser they have a VAE that models the distribution over the joint space and can guide the IK process to stay far away from unplausible poses.

As a tradeoff, these solutions tend to be quite slow, while stuff like pinocchio easily runs in real time.

Do you think predicting additional 3D joint locations is feasible? Or is there a good reason you decided to go with the 15 joint approach instead?

That is a very easy modification to make. If you want to pursue that route, I can provide more details. It mostly depends on changing the keypoint_set used in the FrameDataset here.
The reason I went with the 15 joints is mainly because I had to compare against baselines that were doing the same.

The rest pose skeleton you sent me contains 54 joints and is eerily similar to the SMPLX model, except for the placement of the collar joints [see fig, green circles]. Was this an intentional deviation on your part?

The markerless motion capture software we use here has that skeleton definition, and that's how we get the ground truth data.

Does the dataset you trained FRAME on contain all the 55 joint ground truth locations, so that it would be possible to train on more 3D joint locations?

Yes.

Hope this helps!
Andrea

[kesdogan]

Hi @abcamiletto ,

Thank you so much for your reply! This is super informative.

Just checked the dataset and it even has 69 3D joint positions. I had not seen that before, thank you for pointing me towards it!

I guess, in case one wanted to change this model towards predicting SMPL based relative body pose parameters, the most sensible way forward would be:

1. Derive SMPL pose parameters for the trainings data set, using the existing extensive 3D joint positions in the dataset and some external library (EasyMocap, VPoser, etc.)
2. Modify the current structure of the model to predict relative pose parameters instead of absolute joint locations
3. Retrain model with new dataset

For step (1), the issue with the collar bone joints from my previous image could be solved by interpolating for joints 13, 14 to be between joint 9 and joints 16, 17, and then using the identity rotation for the pose parameter between 13->16, 14->17.

The alternative would be trying to solve the inverse problem using pinocchio for real time performance. However, I could imagine that being jumpy instead, in addition to being a post-processing step that just adds further complexity.

Do you think this is a fair assessment?

Happy to provide more details if you end up going for that direction.

If so, I think it would be super helpful if you could provide more details in that direction!

Regarding EasyMocap, I was not aware that this is able to convert 3D joint positions to SMPL pose parameters. I thought it merely worked with MultiView Images. Is it also able to take 3D joint positions?

Thank you again for the help!!!

Best
Timur

[abcamiletto]

Hi Timur

I guess, in case one wanted to change this model towards predicting SMPL based relative body pose parameters, the most sensible way forward would be:

1. Derive SMPL pose parameters for the trainings data set, using the existing extensive 3D joint positions in the dataset and some external library (EasyMocap, VPoser, etc.)
2. Modify the current structure of the model to predict relative pose parameters instead of absolute joint locations
3. Retrain model with new dataset

Yes, I agree with this. Point 1 is the painful part; I think points 2 and 3 will be smoother sailing.
To be mentioned that another option could be to

1. Modify the current implementation to predict ~55 joints
2. From the 3D predictions, perform IK and obtain SMPL parameters

This might be easier to do (as predicting 55 joints will be mostly straightforward, while predicting SMPL parameters can be tricky sometimes), but it has to be done offline (i.e. save all the predictions from the main model, them convert them to SMPL, then do whatever else).

In both cases, the painful part is the SMPL conversion

For step (1), the issue with the collar bone joints from my previous image could be solved by interpolating for joints 13, 14 to be between joint 9 and joints 16, 17, and then using the identity rotation for the pose parameter between 13->16, 14->17.

The alternative would be trying to solve the inverse problem using pinocchio for real time performance. However, I could imagine that being jumpy instead, in addition to being a post-processing step that just adds further complexity.

Do you think this is a fair assessment?

Yes.

For the record, I think doing it with pinocchio is very challenging: SMPL is very over-parametrized, and doing the IK without a data prior is not an easy task.

Regarding missing or additional joints, imo (but this is debatable), the way to do is to add artificial joints to SMPL in roughly the same position as the current skeleton (this would impact collar bone and hips at least). To do this i would 1) interpolate between smpl existing joints or 2) interpolate between smpl mesh vertices (if you want to have a more granular control).
I did 2) in the past and it works pretty reliably, and gives you fine control over the positioning. Also, a visualizer like this was very helpful in the process.
The operation is differentiable, and you can use these additional joints in the loss function while optimizing SMPL parameters.

Regarding EasyMocap, I was not aware that this is able to convert 3D joint positions to SMPL pose parameters. I thought it merely worked with MultiView Images. Is it also able to take 3D joint positions?

I know of people who modified to work on 3D points. That said, it could be helpful even only for the joint limits they manually computed.

I hope this helps!

Good luck,
Andrea