Inquiry on Real Data Inference & Fine-Tuning Release Timeline

[carolina1998-tech]

Hi,
First, sincere thanks to the team for your valuable open-source contributions—this project has greatly supported my R&D, especially in model optimization.

I have a strong need for real data-based inference and fine-tuning methods in practical scenarios, and are eager to know:

1. Do you have a clear development plan for these methods?
2. Roughly when will they be officially released (or related docs shared)?

I understand open-source timelines depend on multiple factors, Could you kindly share a rough estimate of when real data based might be released?
Thanks again for your work! Looking forward to your reply.

[adithyamurali]

Hi @carolina1998-tech unfortunately I've been swamped to implement this feature... I can give pointers if anyone is interested to implement it.

[Aaryaman3]

Hi @adithyamurali, I'd like to take you up on the offer of pointers.
I've been reading through the code. The dataset format and train.checkpoint are already there, so finetuning infrastructure-wise it feels like most of the plumbing exists. The part I can't quite work out from reading alone is the coordinate frame alignment going from camera/world-frame robot outcomes to the object CoM frame the JSON expects. And how sparse is too sparse when you're working with real labels.
Three things I'd want your input on before starting:

1. Generator, discriminator, or both? With limited real data I'd lean discriminator-only, but I don't know how that interacts with on-generator training.
2. Does finetuning need a mesh, or can it work purely off the point cloud like inference does?
3. Specific robot setup or hardware-agnostic?

My plan would be to start with a tutorial and a conversion script and go from there rather than trying to scope everything upfront.

[adithyamurali]

Hi @Aaryaman3 thank you for that!

• I would first start with finetuning just the discriminator with real world rollouts. If you think about it, this is a form of On-Generator training as well. You'll basically do D grasp rollouts, and K out of D grasps would end up succeeding. You'll have to record the grasp outcomes either checking the positions of the fingertips upon closing/or using force sensors).
• Finetuning can work with either mesh or point clouds. If you have a simulation, you can finetune the model for a new mesh object. If you are doing real world rollouts, you can capture the (segmented) partial view point cloud observations. I feel the latter is more useful
• Technically this is going to be hardware (i.e. gripper) specific. However, since the model itself is object-centric, it will be agnostic to the hardware setup.

You'll need to 1) write your own record script 2) modify the dataloader to accept your new dataset. The training script itself generates a cached dataset (with pre-rendered point clouds) the first time you run it so that it doesn't spend too much time during training to render point clouds. Instead of writing to the original dataset format, it may be better to write to the post-processed cached format, with the real world point clouds, attempted grasps and their binary labels.

Let me know if i can clarify further.