How to annotate keypoints in NDDS in the format specified in DREAM?

[Shashank-Prakash9]

I have installed NDDS but there seems to be no way of marking the keypoints. Any help on this matter will be greatly appreciated

[tabula-rosa]

Hi Shashank, thank you for your interest in DREAM!

For generating the DREAM synthetic datasets, we used the NDDS plug-in for Unreal Engine to export the keypoint information. Unfortunately, I no longer have access to this simulator, and it was not open-sourced for this project, so I am afraid that I can't provide support specifically for NDDS.

I would suggest posting your issue to the NDDS repository (https://github.com/NVIDIA/Dataset_Synthesizer) as they may be better equipped to provide support.

If there are questions about DREAM-specific usage, I might be able to help, but for NDDS more generally, I'm sorry I can't provide further support!

[TontonTremblay]

Yeah we could not make the NDDS + robots public. I would suggest to use nvisii + pybullet. I used it to generate watch it move data. I could probably push a script in the next couple weeks that could generate the right data, but I cannot guarantee it.

[Shashank-Prakash9]

Thank you so much Tabitha and Jonathan. I am thankful for your responses. Jonathan i will definitely explore nvisii and pybullet for the data generation process.My only further query would be , Was the keypoint annotation a feature your team added to NDDS ? or was it already available since I already have a robot model loaded but haven't been able to annotate keypoints.

[TontonTremblay]

It was an added feature for NDDS. Back then we had experts in UE4 helping us building the features we needed. I am afraid that my skills are quite limited to add such feature. But the idea is that you can label a specific point on an asset you want exported in the image space. You easily do that with nvisii and script I wrote.

https://github.com/NVlabs/Deep_Object_Pose/tree/master/scripts/nvisii_data_gen This will get you to about 80% there.

Things to update:

1. load the robot instead of normal objects: https://github.com/NVlabs/Deep_Object_Pose/blob/master/scripts/nvisii_data_gen/single_video_pybullet.py#L410
2. Update the robot pose + nvisii object state: https://github.com/owl-project/NVISII/blob/master/examples/24.urdf.py this is the skeleton on how you can achieve this.
3. Export the joint position keypoints in image space: https://github.com/NVlabs/Deep_Object_Pose/blob/43b685062e79caae921438a220a133895931261c/scripts/nvisii_data_gen/utils.py#L998 This does it for the cuboid around an object. But the way the cuboid is created could simply be hacked to add a keypooint at 0,0,0 of the joint in its local frame than export that single child.

These steps should be somewhat easy to hack. Sorry for vague directions, this assumes that you have quite a bit of 3d knowledge and I am so sorry if it is not the case, this might be quite a bit for someone to debug alone. So if you decide to start in that direction, I would be happy to answer any questions.