2D Keypoints for Each View for Every Frame?

[sshable]

Hi, thanks for the great work!

I'm trying to extract 2D keypoint information for each camera view from the dataset. I looked into the AMASS format and found a pose_body.npy file, which contains 63 keypoints. However, it's not clear what format these keypoints have.

Could you please clarify:

What these 63 keypoints represent?

Whether there are tools or scripts provided in the repo for this purpose?

Any guidance would be greatly appreciated. Thanks!

body_pose = np.load("/home/user/Documents/YogaProject/MvSMPLfitting/amass/moyo_amass/pose_body.npy")
print(body_pose[0])

[-0.24054962 0.00580727 0.005886 -0.13860673 -0.10506066 -0.06160824
0.03851238 0.02816076 0.01131644 0.00317697 -0.24654384 -0.11079165
-0.04728853 0.20157705 0.10608391 -0.38367385 -0.04555607 -0.05854339
-0.00611125 0.31769377 -0.12617444 -0.1264443 -0.37750378 0.23852731
0.15014641 -0.02280479 0.01568092 -0.10524292 -0.04465484 0.10500912
0.11160716 0.15032762 -0.13798463 -0.0340818 -0.05350979 -0.05854491
-0.26495296 0.01853968 0.20493971 -0.18160936 -0.0484682 -0.11197279
0.04113857 0.0798502 0.12885103 -0.13116342 -0.09446818 0.00266627
-0.18328804 -0.00301829 -0.08478944 -0.06476296 0.20477307 -0.25956762
0.022856 -0.10689308 0.2954379 0.13575701 -0.07461731 0.07206708
0.28589204 0.0656374 -0.14308092]

[sha2nkt]

To get 2D keypoints, you will first have to get 3D keypoints. I am assuming you know how to do this but in the MOYO code, we do it here.

Then, to project joints to the image, please refer to Issue #15, especially here. The discussion is about projecting SMPLX vertex mesh on the image but the exact same idea applies to the SMPL joints.

Let me know if you need more help.

[snehashis1997]

hi @sha2nkt thanks for your comment. Can you pls tell me what floating point numbers? Is it 3d points in some kind of normalized version (-1,1)?

[sha2nkt]

These are the 21 body joint locations (x, y, z) flattened. That's why you have 21x3 = 63.

[x1, y1, z1, x2, y2, z2, ... x21, y21, z21]

However, I have not tested projecting the AMASS joints on the image. AMASS uses a slightly different coordinate system so you may have to additionally account for that.

[snehashis1997]

import pickle
import matplotlib.pyplot as plt

Provide the path to your .pkl file

file_path = '/workspaces/ComputerVisionPlayGround/220926_yogi_body_hands_03596_Locust_Pose_or_Salabhasana_-c_stageii.pkl'

Open the file in binary read mode and load the object

with open(file_path, 'rb') as f:
loaded_object = pickle.load(f)

You can now use the loaded_object as needed

print(type(loaded_object))
print(loaded_object.keys())

temp = loaded_object['fullpose'][1]

x = []
y = []
z = []

for i in range(0,165,3):
x.append(temp[i])
y.append(temp[i+1])
z.append(temp[i+2])

x = data[1]['verts'][0][:][0]

y = data[1]['verts'][0][:][0]

z = data[1]['verts'][0][:][0]

Create a 3D scatter plot

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

Scatter plot

scatter = ax.scatter(x, y, z, c=z, cmap='viridis', marker='o')

Add colorbar

fig.colorbar(scatter, ax=ax, label='Z-value')

Set labels

ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_zlabel('Z axis')
ax.set_title('3D Scatter Plot')

Show plot

plt.show()

is this code or plot ok?

[snehashis1997]

this is the animation file for all 1672 frames
https://drive.google.com/file/d/1JVh3MNzaKLdT1pwhEmRR3yulbPm6PhzV/view?usp=sharing