Documentation of the dataset files

[justachetan]

Hi,

Thank you for releasing this amazing work! Could you please provide some additional documentation about the files within the pre-rendered version of the dataset?

Specifically, I am trying to understand what the pose parameters in the bbox files and the metadata files specify?

If I am trying to obtain moving part segmentations from the given dataset, could you please advise on the easiest way to do so?

Thanks again!

[RuiningLi]

Hi,

Thank you for your interest in our work!

Once you un-tarred all files in the pre-rendered version, you can get the moving link IDs for a given episode. Take a look at the code snippet here: https://github.com/RuiningLi/DragAPart/blob/main/dataset.py#L189-L194.

Then, read the instance segmentation for each frame and the moving part segmentation can be obtained by simply instance_segmentation == moving_instance_id.

You can refer to https://github.com/RuiningLi/DragAPart/blob/main/dataset.py as an example.

[justachetan]

Thanks for the pointer @RuiningLi !

I have two quick follow-up questions:

Q1. To the best of my knowledge, it seems that all the continuous joints in the models (like knobs in the "Safe" category or slider buttons in the "WashingMaching" category are not animated independently (as in, they do not move independently of their parent part). Could you kindly confirm whether that is the case?

Q2. From the code that samples drags from the animations, the bounding boxes for each part are oriented bounding boxes obtained from a canonical bounding box. Is that correct?

[RuiningLi]

Hi @justachetan,

For Q1, roughly yes. We only animate parts that demonstrate significant motion, and heuristically we include only parts that are annotated as Hinge Handle, Slider Drawer, Hinge Lid, and Hinge Door by GAPartNet, see Section A.3 here https://arxiv.org/pdf/2403.15382#page=20.20. So the buttons are not animated.

For Q2, correct!

[justachetan]

Hi @RuiningLi!

Thank you for your response! I also had a question regarding shading in the pre-rendered version of the dataset. After examining the images, I noticed that many have flat or no shading (it seems like just the albedo has been rendered).

Furthermore, from the rendering script and the dumped metadata, to render this version you used rasterization instead of ray-tracing.

Q1. Is this flat shading an artifact of using rasterization instead of ray tracing?
Q2. Was there a reason to avoid ray-tracing and instead use rasterization?
Q3. How much time would it take to re-render the dataset using ray-tracing? Or is there a ray-tracing-based version available already?

As always thank you for being so responsive!

[RuiningLi]

Hi @justachetan,

Q1: I believe so!
Q2: The reason is that ray-tracing is about 2-3x slower than rasterization. It took around 5 days to render as I did on about 20 GPUs. I do think there might be room for improvements if you think carefully what and how to save.
Q3: Unfortunately not.

Is the shading very important for what you are working on? If so, I encourage you to render with Blender as the shading there is usually more realistic.

[justachetan]

Hi @RuiningLi,

Thank you for your response! I am indeed interested in getting better shading for the assets during rendering. Is it possible to modify the current rendering script to use Blender instead of Sapien?

I am not too familiar with Sapien but I think one important thing that it does in the script right now is to apply the change of state to the robot's joints (over here). I believe that if I can export the mesh with its textures at this point in the pipeline and provide its reference to Blender, then it should be possible to render the scene in Blender with minimal modifications to the remaining pipeline.

Do you think this plan makes sense? Is there a way to export the mesh along with textures once the changes to the joint positions have been applied?

As always, thank you for your continued help!

Happy New Year!

Regards,
Aditya

[RuiningLi]

The assets in Sapien is stored in URDF format, which can be easily exported to meshes. You might need some care to ensure the texture is correctly exported though.