Locomanipulation with Improved IK Control, G1 teleoperation and Locomotion policy #1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
d2e5558 to
6951dbf
Compare
michaellin6
force-pushed
the
mlin/teleop-ik-improv
branch
2 times, most recently
from
2affaeb to
e3940bc
Compare
michaellin6
force-pushed
the
mlin/teleop-ik-improv
branch
3 times, most recently
from
b240525 to
2ea0e0d
Compare
…ia pip (isaac-sim#3225) # Description If Isaac Lab is installed through pip, tasks generated with the Internal template cannot be placed inside the site-packages directory of that installation. Since this setup would never work correctly, it makes no sense to offer the Internal template as an option. This PR therefore disables that option entirely whenever Isaac Lab is detected as running from a pip-installed path. Fixes # (issue) ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - This change requires a documentation update <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
This PR fixes typo in isaaclab.bat Should be `!allArgs!` instead of `!allArgs1` ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
…v… (isaac-sim#3149) # Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Enhance Pink IK Controller with Null Space Posture Control This PR improves the Pink IK controller integration for better humanoid robot control and more natural postures. **Note**: Original this PR was staged in the internal repo (isaac-sim#547). It has been moved here due to new Github workflow. ## Key Changes ### New Null Space Posture Task - Added NullSpacePostureTask to enforce postural constraints on shoulder/waist joints while prioritizing end-effector tasks - Maintains natural robot poses during manipulation ### Controller Improvements - Tuned low level PD controller gains - Support mixed task types (FrameTask + NullSpacePostureTask) ### Testing & Environment Updates - Redesigned pink controller test script to use JSON-based configurations to program test motions. - Updated all environments (PickPlace, NutPour, ExhaustPipe) with null space control, damping tasks, and improved tracking - Added `Isaac-PickPlace-GR1T2-WaistEnabled-Abs-v0` env that is identical to `Isaac-PickPlace-GR1T2-Abs-v0` but enables the Waist DOFs. - Added target_eef_link_names mapping for clearer link specification Fixes # (issue) These changes help fix the following problems from [VDR feedback](https://docs.google.com/document/d/1saB1QA5r_WlD1l17q7C04WWNltnBW-K0ydI2UB8jxAs/edit?tab=t.0) - [Enable Waist DOF](https://nvbugspro.nvidia.com/bug/5235527) - Discourage elbow flare - Make controller low-latency and low-jerk. **We improved the unit test for the pink controller and reduced our position and rotation accuracy tolerance from 30 mm, 10 degrees to 1 mm, 1 degree.** - Develop metric for controller performance - Added a flag to disable failure due to joint limits. Previously, any commanded pose that ended in joint limit violation would result in no solution and the controlled robot freezing in place. This change gets the solver to still provide a solution and instead issue a warning for joint limit violations. ## Screenshots These controller changes have been tested through the Mimic pipeline (teleop_se3_agent.py, record/replay_demos.py). Here are videos showing teleoperation of all three environments working. ### PickPlace-GR1T2-Abs  ### NutPour-GR1T2  ### ExhaustPipe-GR1T2  ### Successfully Trained Robomimic Model Rollout on PickPlace task For the two robomimic tasks: `Isaac-PickPlace-GR1T2-Abs-v0` and `Isaac-NutPour-GR1T2-Pink-IK-Abs-v0`, if we collect a new dataset, we achieve a success rate of 96 and 92% respectively.   ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
# Description Corrected the file path for the list_envs.py script in the documentation. Fixes # (issue) ## Type of change - Fix a typo ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there Signed-off-by: Francisco Beltrao <fbeltrao@windowslive.com>
# Description We have been hunting down a strange issue in distributed training setups with rendering enabled, where often the process would hang midway through training and causes NCCL timeouts. A workaround was discovered to set `app.execution.debug.forceSerial = true`, which forces serialized scheduling of omni graph within the same thread. This appears to have resolved the hanging issue and did not cause performance regressions. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
# Description The test priniting, sometimes gets super verbose and non-readable when isaacsim or kit are in bad state. since we only care about failure test cases and those printing, we can disable --verbose flag ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Kelly Guo <kellyguo123@hotmail.com>
…port error (isaac-sim#3292) # Description Feature introduced in isaac-sim#3149 imports pink_ik for isaaclab.controller module. This is causing an error IsaacLab wide due to pinocchio import. This PR removes the import. Fixes # (issue) Fixed pinocchio import error due to isaaclab.controller module importing pink_ik by default. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Breaking change (fix or feature that would cause existing functionality to not work as expected) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
…-sim#3180) # Description Fix terrain_out_of_bounds to return tensor instead of bool Ensure the function always returns a PyTorch tensor (shape [num_envs]) rather than a Python boolean when terrain_type is "plane", preventing AttributeError in termination_manager. Before: ```bash Error executing job with overrides: [] Traceback (most recent call last): File "/home/ubuntu/workspaces/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py", line 101, in hydra_main func(env_cfg, agent_cfg, *args, **kwargs) File "/home/ubuntu/workspaces/robot_lab/scripts/reinforcement_learning/rsl_rl/train.py", line 165, in main runner.learn(num_learning_iterations=agent_cfg.max_iterations, init_at_random_ep_len=True) File "/home/ubuntu/miniconda3/envs/lab/lib/python3.11/site-packages/rsl_rl/runners/on_policy_runner.py", line 206, in learn obs, rewards, dones, infos = self.env.step(actions.to(self.env.device)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/workspaces/IsaacLab/source/isaaclab_rl/isaaclab_rl/rsl_rl/vecenv_wrapper.py", line 176, in step obs_dict, rew, terminated, truncated, extras = self.env.step(actions) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/miniconda3/envs/lab/lib/python3.11/site-packages/gymnasium/wrappers/common.py", line 393, in step return super().step(action) ^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/miniconda3/envs/lab/lib/python3.11/site-packages/gymnasium/core.py", line 327, in step return self.env.step(action) ^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/workspaces/IsaacLab/source/isaaclab/isaaclab/envs/manager_based_rl_env.py", line 204, in step self.reset_buf = self.termination_manager.compute() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ubuntu/workspaces/IsaacLab/source/isaaclab/isaaclab/managers/termination_manager.py", line 172, in compute rows = value.nonzero(as_tuple=True)[0] # indexing is cheaper than boolean advance indexing ^^^^^^^^^^^^^ AttributeError: 'bool' object has no attribute 'nonzero' ``` ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com>
# Description Temporary disabled test reporting for PRs created from the forks.
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
6951dbf to
94e6cc1
Compare
…m#3283) # Description There have been two places where the template documentation has been placed (under Developers Guide and Workthrough), this PR unifies them into a new structure (see image below). Furthermore, the imitation learning examples were missing a grouping, this PR introduces a structure similar to the section about reinforcement learning Also some general docs fixes are included. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots <img width="588" height="1323" alt="image" src="https://github.com/user-attachments/assets/a17a6328-a9e3-44cd-a299-6aa62e0e422e" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
# Description Updates version of the framework for 2.2.1 patch release. ## Type of change - This change requires a documentation update ## Checklist - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [ ] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
# Description `GroundPlaneCfg` allows for specifying `visible` parameter, but this would not being parsed in `spawn_ground_plane`, resulting in the parameter being a no-op when specified. This change adds a fix to parse the `visible` parameter from the cfg and sets the visibility attribute for the ground plane cfg appropriately. Fixes isaac-sim#3263 <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
# Description This PR adds the necessary changes to work with the new version of RSL RL. ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Clemens Schwarke <96480707+ClemensSchwarke@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Octi Zhang <zhengyuz@nvidia.com>
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> This PR: - Add task_runner.py to support specifying resources, py_modules, and pip. Fixes [# (issue)](isaac-sim#2632) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: garylvov <67614381+garylvov@users.noreply.github.com> Co-authored-by: 松翊 <songyi.wb@alibaba-inc.com> Co-authored-by: garylvov <67614381+garylvov@users.noreply.github.com>
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
2 times, most recently
from
54cdf26 to
15ee2d7
Compare
# Description This PR 1. makes sure(skip if already satisfy, else install) the right torch is installed before and after pip installing isaaclab packages as sometime(rare case) due to flaky setup.py and unknown library dependencies changes pytorch version gets overriden. 2. only install pink and retargeters in linux x86 or amd64 machines ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com>
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
2 times, most recently
from
4f82310 to
2ba1a9e
Compare
…pper (isaac-sim#3327) Fixes the reach task regression with teleop devices returning the gripper term # Description Fixes the reach task regression with teleop devices returning the gripper. The reach task expects just the se3 term and not the gripper term. We add a configuration parameter to the teleop devices which do not use retargeters to conditional return the gripper term, and update the reach env cfg to properly configure the teleop devices. <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. Fixes isaac-sim#3264 <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rwiltz <165190220+rwiltz@users.noreply.github.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
…ration (isaac-sim#3316) # Description This PR fixes a bug where the platform_height parameter was incorrectly placed in the MeshPyramidStairsTerrainCfg class instead of the appropriate base configuration class for mesh terrain objects. - Removes the misplaced `platform_height` parameter from `MeshPyramidStairsTerrainCfg` - Adds the `platform_height` parameter to the correct location in the `MeshRepeatedObjectsTerrainCfg` class - Includes various formatting improvements with additional blank lines for consistency Fixes isaac-sim#3162 Regression was introduced in MR isaac-sim#2695 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
769ade1 to
188e3d7
Compare
# Description Update locomanip task name and link in docs ## Type of change - This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
188e3d7 to
b605c6f
Compare
Updated GLIBC version requirement for pip installation. # Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. Fixes the documentation by taking the info from https://docs.isaacsim.omniverse.nvidia.com/5.0.0/installation/install_python.html <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - A small fix in the documentation ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> Signed-off-by: Giulio Romualdi <giulio.romualdi@gmail.com>
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> We only have a surface gripper sample with direct workflow, whereas in manager-based workflow it is not supported yet. - Add surface gripper as an asset instance to the scene (CPU only) - Add SurfaceGripperAction and SurfaceGripperActionCfg - Add two TaskEnvs for testing: 1. Isaac-Stack-Cube-UR10-Long-Suction-IK-Rel-v0 2. Isaac-Stack-Cube-UR10-Short-Suction-IK-Rel-v0 You can test recording demos by: `./isaaclab.sh -p scripts/tools/record_demos.py --task Isaac-Stack-Cube-UR10-Long-Suction-IK-Rel-v0 --teleop_device keyboard --device cpu` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshot <img width="1386" height="273" alt="environments_surface_gripper" src="https://github.com/user-attachments/assets/4c1cb5a4-4325-4861-b20c-9204e98e7dff" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
# Description Recent changes introduced a minor bug: now, Ray is not initialized when `tuner.py` is called. This PR fixes this by adding back the removed initialization. Fixes isaac-sim#3349 ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Screenshots <img width="1637" height="34" alt="pr_changes" src="https://github.com/user-attachments/assets/820c55ce-9951-405a-ac7b-005dfeab877c" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
a87b280 to
ebbea14
Compare
# Description Since suction gripper requires CPU simulation currently, we disable GPU environment testing for them for now and explicitly sets the device for these environments to CPU. ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
# Description Another time of manually fixing errors seen in the docs. We should have CI strictly enforce doc build warnings so they get removed before MR is merged. ## Type of change - This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
) # Description Earlier, the function `check_usd_path_with_timeout` was in `sim/utils.py` while all file related operations live in `utils/asset.py`. This MR moves the function to the right location. Fixes # (issue) ## Type of change - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Co-authored-by: Kelly Guo <kellyg@nvidia.com>
# Description Deleted the `utils` folder which should not exist ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
20a558d to
f8b0a43
Compare
# Description Support getting hand tracking data from manus gloves (joint poses relative to wrists) and vive trackers (wrist poses, calibrated with AVP wrist poses). ## Type of change - New feature (non-breaking change which adds functionality) ## Checklist - [ x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ x] I have made corresponding changes to the documentation - [ x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
# Description Updates Isaac Sim license to the latest Apache 2.0 license and fixes broken link in the docs. ## Type of change - This change requires a documentation update
# Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Adds two new robots with grippers: - franka + robotiq_2f_85 gripper, with a new usd asset - ur10e + robotiq_2f_140 gripper, with IsaacSim UR10E asset with variants Test the two new arms with gripper with this script: [test_new_arms.py](https://github.com/user-attachments/files/22200295/test_new_arms.py) `./isaaclab.sh -p test_new_arms.py` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - New feature (non-breaking change which adds functionality) ## Screenshots  <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
68b387b to
5bd7cba
Compare
…on (isaac-sim#3356) # Description 1. Add a notification widget when ik error happens 2. At the end of Teleop data collection, display a notification before the application termination <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change. Fixes # (issue) <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) - Breaking change (fix or feature that would cause existing functionality to not work as expected) - This change requires a documentation update ## Screenshots Please attach before and after screenshots of the change if applicable. <!-- Example: | Before | After | | ------ | ----- | | _gif/png before_ | _gif/png after_ | To upload images to a PR -- simply drag and drop an image while in edit mode and it should upload the image directly. You can then paste that source into the above before/after sections. --> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
…tion, using RMPFlow controller (isaac-sim#3210) # Description <!-- Thank you for your interest in sending a pull request. Please make sure to check the contribution guidelines. Link: https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html --> Adds galbot_stack_cube tasks and mimic tasks, using RMPFlow controller: - add galbot robot config and .usd asset - add RMPFlowAction and RMPFlowActionCfg - add motion_policy_configs and .urdf for both 'galbot_left_arm_gripper' and 'galbot_right_arm_suction' - add new task: galbot stack_rmp_rel_env_cfg - add new mimic task: galbot_stack_rmp_abs/rel_mimic_env - add mdp.observations/terminations/events for galbot: support gripper_state checking for both suction_cup and parallel_gripper, get obs_in_base_frame - add gripper_configs (gripper_joint_names, gripper_open_val, gripper_threshold) in galbot/franka tasks: to make mdp functions universal to varied robots - fix a bug (eef_name) in franka_stack_ik_rel_mimic_env.py - add new device_name in se3_spacemouse.py Notes: This PR relies on PR (isaac-sim#3174) for surface gripper support in manager-based workflow. You can test the whole gr00t-mimic workflow by: 1. record demos: `./isaaclab.sh -p scripts/tools/record_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-v0 --teleop_device spacemouse --num_demos 1 --device cpu --dataset_file datasets/recorded_demos_galbot_suction_rel.hdf5` 2. replay demos: `./isaaclab.sh -p scripts/tools/replay_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-v0 --num_envs 1 --device cpu --dataset_file datasets/recorded_demos_galbot_suction_rel.hdf5` 3. annotate demos: `./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-Rel-Mimic-v0 --auto --device cpu --input_file datasets/recorded_demos_galbot_suction_rel.hdf5 --output_file datasets/annotated_demos_galbot_suction_rel.hdf5` 4. generate dataset: `./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py --task Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-Rel-Mimic-v0 --num_envs 16 --device cpu --generation_num_trials 10 --input_file datasets/annotated_demos_galbot_suction_rel.hdf5 --output_file datasets/generated_demos_galbot_suction_rel.hdf5` <!-- As a practice, it is recommended to open an issue to have discussions on the proposed pull request. This makes it easier for the community to keep track of what is being developed or added, and if a given feature is demanded by more than one party. --> ## Type of change <!-- As you go through the list, delete the ones that are not applicable. --> - Bug fix (non-breaking change which fixes an issue) - New feature (non-breaking change which adds functionality) ## Screenshot <img width="1428" height="281" alt="environments_galbot" src="https://github.com/user-attachments/assets/b603bc60-7ff3-44e8-bf41-ab8126a753ec" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task --> --------- Signed-off-by: rebeccazhang0707 <168459200+rebeccazhang0707@users.noreply.github.com> Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com>
) ## Description This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility. **Note:** - Please look at the cuRobo usage license  - Please look at updated isaacsim license  ### Technical Implementation: **Annotation Framework:** - Manual subtask start annotations to cleanly separate skill execution from motion-planning segments - Consistent trajectory segmentation for downstream dataset consumers **Motion Planning:** - **Base Motion Planner (Extensible):** - Introduces a reusable planner interface for uniform integration: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py` - Defines a minimal, consistent API for planners: - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc. - The cuRobo planner inherits from this base class. - New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline. - **CuRobo Planner** (GPU-accelerated, collision-aware): - Multi-phase planning: approach → contact → retreat - Dynamic object attach/detach and contact-aware sphere management - Real-time world synchronization between Isaac Lab and cuRobo - Configurable collision filtering for contact phases - **Tests**: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py` - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py` - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py` **Data Generation Pipeline:** - Automated dataset generation with precise skill-based segmentation - Integrates with existing observation/action spaces - Supports multi-env parallel collection with cuRobo-backed planning **Visualization and Debugging:** - Rerun-based 3D visualization for trajectory/collision inspection - Real-time sphere visualization for collision boundaries and contact phases ### Dependencies: - **cuRobo**: motion planning and collision checking - **Rerun**: 3D visualization and debugging ### Integration: This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users. ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] This change requires a documentation update ## Screenshot ### SkillGen Data Generation <table> <tr> <td align="center"><strong>Cube Stacking SkillGen Data Generation</strong></td> <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae" alt="Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e" alt="Bin Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ### Bin Cube Stacking Behavior Cloned Policy  ### Rerun Integration  ### Motion Planner Tests <table> <tr> <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td> <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db" alt="Obstacle Avoidance cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde" alt="Cube Stack End-to-End cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Kelly Guo <kellyg@nvidia.com> Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com>
… support - Implement fixed-base upper body IK and locomanipulation environments - Add G1 teleop with retargeter, hand rotation fixes, and comprehensive tests - Refactor locomotion to use retargeter instead of command manager - Enhance pink IK with kinematics model and LocalFrameTask for relative poses - Add locomanipulation policies with gravity compensation and tuned gains - Implement lower body standing retargeter with zero root velocity Co-authored-by: Michael Lin <michalin@nvidia.com> Co-authored-by: Huihua Zhao <huihuaz@nvidia.com> Co-authored-by: Rafael Wiltz <rwiltz@nvidia.com> Co-authored-by: Sergey Grizan <sgrizan@nvidia.com>
… import error. Also increasing pink_ik test timeout
…m#3392) # Description Recent isaac sim update introduced a new bug for non-headless scripts where some scripts were hanging at simulation startup. This change introduces a new unit test that aims to capture issues like this by forcing the use of the non-headless app file. Additionally, the isaac sim CI system has very unstable results for perf testing, so we are disabling the performance-related tests for the sim CI. ## Type of change - Bug fix (non-breaking change which fixes an issue) ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there <!-- As you go through the checklist above, you can mark something as done by putting an x character in it For example, - [x] I have done this task - [ ] I have not done this task -->
…ac-sim#3405) # Description The default_inertia attributes of the Articulation, RigidObjectCollection, and RigidObject data asset classes did not specify in what coordinate frame the tensors should be provided. This PR addresses this, and addresses some minor inconsistencies across the default_inertia docstrings. ## Type of change - This change requires a documentation update ## Screenshots ArticulationData | Before | After | | ------ | ----- | | <img width="731" height="162" alt="image" src="https://github.com/user-attachments/assets/7e2a6973-6264-4ba0-89d7-fd903e3bee6d" />| <img width="750" height="262" alt="image" src="https://github.com/user-attachments/assets/470832e6-0164-4402-b4c7-6a5ac5e599a1" />| RigidObjectCollectionData | Before | After | | ------ | ----- | | <img width="731" height="181" alt="image" src="https://github.com/user-attachments/assets/7d6bd039-b7f5-40e2-9180-33d0748694a8" /> | <img width="733" height="276" alt="image" src="https://github.com/user-attachments/assets/07bb32ca-628c-4132-8009-1db1c279d653" /> | RigidObjectData | Before | After | | ------ | ----- | | <img width="732" height="160" alt="image" src="https://github.com/user-attachments/assets/43b29f1c-3e92-4b96-925f-88cdf7ef3fc3" /> | <img width="733" height="276" alt="image" src="https://github.com/user-attachments/assets/d16bbc57-3939-4f2e-84cb-9ec135dcbe7e" /> | ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there
michaellin6
force-pushed
the
team/locomanipulation-integration
branch
from
048daf0 to
a9afae2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Contributors: @huihuaNvidia2023, @rwiltz, @life1ess, @peterd-NV, @yanchangNvidia
These changes introduce new environments:
Isaac-PickPlace-Locomanipulation-G1-Abs-v0andIsaac-PickPlace-FixedBaseUpperBodyIK-G1-Abs-v0. These environments, specially the locomanipulation one, feature:Below is a diagram showing the high level controller composition. The environments introduced have the waist controlled by IK, but it is possible to configure it such that waist DOFs are controlled by the lower body policy as well.
This code has been tested through the Mimic pipeline, and the output dataset was used to train a robomimic policy that achieves a success rate of 92% from 50 policy rollouts.
Fixes # (issue)
Some fixes included in these changes:
class:PinkKinematicsConfigurationthat allows for all joints of an articulation to be updated while only a subset are controlled by IK.Type of change
Screenshots
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there