feat: real-world discriminator fine-tuning pipeline (closes #29)#56
Open
Aaryaman3 wants to merge 2 commits intoNVlabs:mainfrom
Open
feat: real-world discriminator fine-tuning pipeline (closes #29)#56Aaryaman3 wants to merge 2 commits intoNVlabs:mainfrom
Aaryaman3 wants to merge 2 commits intoNVlabs:mainfrom
Conversation
- GraspRecorder: records grasp attempts (pose, point cloud, confidence, success, collision) to rollouts.h5 + grasps.json - write_cache(): converts recordings to GraspGenDatasetCache HDF5 format for preload_dataset=True training; appends to file instead of clobbering - from_h5(): reconstructs GraspRecorder from saved files (used by prepare_finetune.py) - prepare_finetune.py: user-facing script that validates rollout data, builds the training cache, and prints the exact fine-tune command - 20 unit tests covering save/load, labels, round-trip, multi-object cache, and config key validation - dataset.py fixes: numpy→tensor guard for preloaded point clouds, meshless guard when no .obj file exists Fine-tuning signal validated on GCP VM: discriminator pos/neg gap increased from -0.0012 → +0.0155 after 50 steps on trimesh-labeled banana grasps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds infrastructure for fine-tuning the GraspGen discriminator on real-world robot rollout data (closes #29).
New files
scripts/record_grasp.py—GraspRecorder: records grasp attempts (pose, point cloud, confidence, success/fail, collision) torollouts.h5+grasps.json. Also writes directly toGraspGenDatasetCacheHDF5 viawrite_cache().scripts/prepare_finetune.py— CLI: validates rollout data, builds the training cache, prints the exacttrain_graspgen.pycommand to run.tests/test_record_grasp.py— 20 unit tests: save/load, label correctness,from_h5()round-trip, multi-object cache writes, config key validation.scripts/test_training_step.py— integration tests for a real discriminator training step (with and without mesh) on the GCP VM.Bug fixes in
dataset.pypreload_dataset=True)load_discriminator_batch_with_stratified_samplingwhen no.objfile existsThese two were breaking real-world use no matter what else you did right.
Validated on GCP VM
Discriminator pos/neg gap: -0.0012 → +0.0155 after 50 fine-tune steps on trimesh-labeled banana grasps.
Not covered
discriminator_ratiomust be 7-element with on-policy data (documented inrecord_grasp.pyheader).Test plan
pytest tests/test_record_grasp.py— all 20 passpython scripts/test_training_step.py—run()andrun_meshless()both pass on the VMpython scripts/prepare_finetune.py --help— args parse correctly