What pose is being outputted in pnp_results.csv?

[andrewjjeon]

Hello, I have run

python scripts/network_inference_dataset.py -i trained_models/panda_dream_vgg_q.pth -d data/real/panda-3cam_realsense/ -o outputs -b 16 -w 8

and I am now trying to compare the "predicted pose" which I believe is in pnp_results.csv which was generated from the run.

I'm comparing the x y z predicted positions

with the ground truth positions in the .json annotations for each frame.

What pose is being recorded in pnp_results.csv? It doesn't seem to match well with link0 pose or "end-effector/hand" pose, so I am confused.

[tabula-rosa]

Hi Andrew, thanks for your interest in DREAM!

Apologies for the delay in my response. Did you figure out the coordinate frames or is there a way I can still help?

If I recall correctly, in the json file, the coordinate frame for the location fields of the keypoints is with respect to the camera. So, the third entry of location, the z-position, should always be positive.

For the csv file, the poses are the translation and quaternion results from dream.geometric_vision.solve_pnp which uses cv2.solvePnP under the hood. From re-reading the code and my own recollection, I believe the coordinate frame being used for the 3D positions of the keypoints kp_pos_gt_pnp here is the camera frame. So the resulting pose from solve_pnp is the pose difference between the detected camera frame and the ground truth camera frame. This pose difference is then used to calculate the ADD metric.

Let me know if that helps or if that differs from your conclusions or understanding!