Skip to content

merge in isaaclab main #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
peterd-NV wants to merge 9 commits into from
Closed

merge in isaaclab main #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
3a0db9d
Updates default viewer pose to see the whole scene for Agibot environ…
rebeccazhang0707 Sep 22, 2025
cd204ad
Adds torch hook to export libgomp.so.1 from installed torch (#3512)
ooctipus Sep 24, 2025
bd547aa
Adds instructions on how to position the lighthouse for manus+vive (#…
cathyliyuanchen Sep 25, 2025
0a417e8
[Newton] Updates Newton docs for beta integration (#3518)
kellyguo11 Sep 28, 2025
34729a3
[Newton] Updates newton visualizer documentation (#3551)
Milad-Rakhsha-NV Sep 28, 2025
faa96df
Provides PBT training cfg example for Isaac-Dexsuite-Kuka-Allegro-Lif…
ooctipus Sep 29, 2025
902bded
[Newton] Adds policy transfer script for sim2sim transfer from Newton…
Milad-Rakhsha-NV Sep 29, 2025
21bcb47
Randomizes viscous and dynamic joint friction based on IsaacSim 5.0 (…
GiulioRomualdi Sep 29, 2025
ad4ce34
Fixes broken link to conda in installation doc (#3601)
mpgussert Oct 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 27 additions & 7 deletions ...experimental-features/newton-physics-integration/limitations-and-known-bugs.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,28 +9,48 @@ We do not expect to be able to provide support or debugging assistance until the
Here is a non-exhaustive list of capabilities currently supported in the Newton experimental feature branch grouped by extension:

* isaaclab:
* Articulation API
* Articulation API (supports both articulations and single-body articulations as rigid bodies)
* Contact Sensor
* Direct & Manager single agent workflows
* Omniverse Kit visualizer
* Newton visualizer
* isaaclab_assets:
* Anymal-D
* Unitree H1 & G1
* Quadrupeds
* Anymal-B, Anymal-C, Anymal-D
* Unitree A1, Go1, Go2
* Spot
* Humanoids
* Unitree H1 & G1
* Cassie
* Arms and Hands
* Franka
* UR10
* Allegro Hand
* Toy examples
* Cartpole
* Ant
* Humanoid
* isaaclab_tasks:
* Direct:
* Cartpole
* Cartpole (State, RGB, Depth)
* Ant
* Humanoid
* Allegro Hand Repose Cube
* Manager based:
* Cartpole (State)
* Ant
* Humanoid
* Locomotion (velocity flat terrain)
* Anymal-B
* Anymal-C
* Anymal-D
* Cassie
* A1
* Go1
* Go2
* Spot
* Unitree G1
* Unitree H1

Capabilities beyond the above are not currently available.
We expect to support APIs related to rigid bodies soon in order to unlock manipulation based environments.
* Manipulation reach
* Franka
* UR10
6 changes: 5 additions & 1 deletion docs/source/experimental-features/newton-physics-integration/newton-visualizer.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,4 +32,8 @@ lower number of environments, we can omit the ``--headless`` option while still

These options are available across all the learning frameworks.

For more information about the Newton Visualizer, please refer to the `Newton documentation <https://newton-physics.github.io/newton/guide/visualization.html>`_ .
For more information about the Newton Visualizer, please refer to the `Newton documentation <https://newton-physics.github.io/newton/guide/visualization.html>`_.

IsaacLab provides additional customizations to the Newton visualizer with several learning-oriented features. These include the ability to pause rendering during training or pause the training process itself. Pausing rendering accelerates training by skipping rendering frames, which is particularly useful when we want to periodically check the trained policy without the performance overhead of continuous rendering. Pausing the training process is valuable for debugging purposes. Additionally, the visualizer's update frequency can be adjusted using a slider in the visualizer window, making it easy to prioritize rendering quality against training performance and vice-versa.

All IsaacLab-specific customizations are organized under the *IsaacLab Training Controls* tab in the visualizer window.
114 changes: 91 additions & 23 deletions docs/source/experimental-features/newton-physics-integration/sim-to-sim.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,55 @@

Sim-to-Sim Policy Transfer
==========================
This section provides examples of sim-to-sim policy transfer using the Newton backend. Sim-to-sim transfer is an essential step before real robot deployment because it verifies that policies work across different simulators. Policies that pass sim-to-sim verification are much more likely to succeed on real robots.
This section provides examples of sim-to-sim policy transfer between PhysX and Newton backends. Sim-to-sim transfer is an essential step before real robot deployment because it verifies that policies work across different simulators. Policies that pass sim-to-sim verification are much more likely to succeed on real robots.


Overview
--------

This guide shows how to run a PhysX-trained policy on the Newton backend. While the method works for any robot and physics engine, it has only been tested with Unitree G1, Unitree H1, and ANYmal-D robots using PhysX-trained policies.
This guide shows how to transfer policies between PhysX and Newton backends in both directions. The main challenge is that different physics engines may parse the same robot model with different joint and link ordering.

PhysX-trained policies expect joints and links in a specific order determined by how PhysX parses the robot model. However, Newton may parse the same robot with different joint and link ordering.
Policies trained in one backend expect joints and links in a specific order determined by how that backend parses the robot model. When transferring to another backend, the joint ordering may be different, requiring remapping of observations and actions.

In the future, we plan to solve this using **robot schema** that standardizes joint and link ordering across different backends.

Currently, we solve this by remapping observations and actions using joint mappings defined in YAML files. These files specify joint names in both PhysX order (source) and Newton order (target). During policy execution, we use this mapping to reorder observations and actions so they work correctly with Newton.
Currently, we solve this by remapping observations and actions using joint mappings defined in YAML files. These files specify joint names in both source and target backend orders. During policy execution, we use this mapping to reorder observations and actions so they work correctly with the target backend.

The method has been tested with Unitree G1, Unitree Go2, Unitree H1, and ANYmal-D robots for both transfer directions.


What you need
~~~~~~~~~~~~~

- A policy checkpoint trained with PhysX (RSL-RL).
- A joint mapping YAML for your robot under ``scripts/newton_sim2sim/mappings/``.
- The provided player script: ``scripts/newton_sim2sim/rsl_rl_transfer.py``.
- A policy checkpoint trained with either PhysX or Newton (RSL-RL).
- A joint mapping YAML for your robot under ``scripts/sim2sim_transfer/config/``.
- The provided player script: ``scripts/sim2sim_transfer/rsl_rl_transfer.py``.

To add a new robot, create a YAML file with two lists where each joint name appears exactly once in both:

.. code-block:: yaml

# Example structure
source_joint_names: # PhysX joint order
source_joint_names: # Source backend joint order
- joint_1
- joint_2
# ...
target_joint_names: # Newton joint order
target_joint_names: # Target backend joint order
- joint_1
- joint_2
# ...

The script automatically computes the necessary mappings for locomotion tasks.


How to run
~~~~~~~~~~
PhysX-to-Newton Transfer
~~~~~~~~~~~~~~~~~~~~~~~~

Use this command template to run a PhysX-trained policy with Newton:
To run a PhysX-trained policy with the Newton backend, use this command template:

.. code-block:: bash

./isaaclab.sh -p scripts/newton_sim2sim/rsl_rl_transfer.py \
./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=<TASK_ID> \
--num_envs=32 \
--checkpoint <PATH_TO_PHYSX_CHECKPOINT> \
Expand All @@ -60,40 +62,106 @@ Here are examples for different robots:

.. code-block:: bash

./isaaclab.sh -p scripts/newton_sim2sim/rsl_rl_transfer.py \
./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-G1-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_PHYSX_CHECKPOINT> \
--policy_transfer_file scripts/newton_sim2sim/mappings/sim2sim_g1.yaml
--policy_transfer_file scripts/sim2sim_transfer/config/physx_to_newton_g1.yaml


2. Unitree H1


.. code-block:: bash

./isaaclab.sh -p scripts/newton_sim2sim/rsl_rl_transfer.py \
./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-H1-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_PHYSX_CHECKPOINT> \
--policy_transfer_file scripts/newton_sim2sim/mappings/sim2sim_h1.yaml
--policy_transfer_file scripts/sim2sim_transfer/config/physx_to_newton_h1.yaml


3. Unitree Go2

.. code-block:: bash

3. ANYmal-D
./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-Go2-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_PHYSX_CHECKPOINT> \
--policy_transfer_file scripts/sim2sim_transfer/config/physx_to_newton_go2.yaml


4. ANYmal-D


.. code-block:: bash

./isaaclab.sh -p scripts/newton_sim2sim/rsl_rl_transfer.py \
./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-Anymal-D-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_PHYSX_CHECKPOINT> \
--policy_transfer_file scripts/newton_sim2sim/mappings/sim2sim_anymal_d.yaml
--policy_transfer_file scripts/sim2sim_transfer/config/physx_to_newton_anymal_d.yaml

Note that to run this, you need to checkout the Newton-based branch of IsaacLab such as ``feature/newton``.

Newton-to-PhysX Transfer
~~~~~~~~~~~~~~~~~~~~~~~~

To transfer Newton-trained policies to PhysX-based IsaacLab, use the reverse mapping files:

Here are examples for different robots:

1. Unitree G1

.. code-block:: bash

./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-G1-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_NEWTON_CHECKPOINT> \
--policy_transfer_file scripts/sim2sim_transfer/config/newton_to_physx_g1.yaml


2. Unitree H1

.. code-block:: bash

./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-H1-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_NEWTON_CHECKPOINT> \
--policy_transfer_file scripts/sim2sim_transfer/config/newton_to_physx_h1.yaml


3. Unitree Go2

.. code-block:: bash

./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-Go2-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_NEWTON_CHECKPOINT> \
--policy_transfer_file scripts/sim2sim_transfer/config/newton_to_physx_go2.yaml


4. ANYmal-D

.. code-block:: bash

./isaaclab.sh -p scripts/sim2sim_transfer/rsl_rl_transfer.py \
--task=Isaac-Velocity-Flat-Anymal-D-v0 \
--num_envs=32 \
--checkpoint <PATH_TO_NEWTON_CHECKPOINT> \
--policy_transfer_file scripts/sim2sim_transfer/config/newton_to_physx_anymal_d.yaml

The key difference is using the ``newton_to_physx_*.yaml`` mapping files instead of ``physx_to_newton_*.yaml`` files. Also note that you need to checkout a PhysX-based IsaacLab branch such as ``main``.

Notes and limitations
Notes and Limitations
~~~~~~~~~~~~~~~~~~~~~

- This transfer method has only been tested with Unitree G1, Unitree H1, and ANYmal-D using PhysX-trained policies.
- The observation remapping assumes a locomotion layout with base observations followed by joint observations. For different observation layouts, you'll need to modify ``scripts/newton_sim2sim/policy_mapping.py``.
- Both transfer directions have been tested with Unitree G1, Unitree Go2, Unitree H1, and ANYmal-D robots.
- PhysX-to-Newton transfer uses ``physx_to_newton_*.yaml`` mapping files.
- Newton-to-PhysX transfer requires the corresponding ``newton_to_physx_*.yaml`` mapping files and the PhysX branch of IsaacLab.
- The observation remapping assumes a locomotion layout with base observations followed by joint observations. For different observation layouts, you'll need to modify the ``get_joint_mappings`` function in ``scripts/sim2sim_transfer/rsl_rl_transfer.py``.
- When adding new robots or backends, make sure both source and target have identical joint names, and that the YAML lists reflect how each backend orders these joints.
16 changes: 16 additions & 0 deletions ...urce/experimental-features/newton-physics-integration/training-environments.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,28 @@ To run training, we follow the standard Isaac Lab workflow. If you are new to Is
The currently supported tasks are as follows:

* Isaac-Cartpole-Direct-v0
* Isaac-Cartpole-RGB-Camera-Direct-v0 (requires ``--enable_cameras``)
* Isaac-Cartpole-Depth-Camera-Direct-v0 (requires ``--enable_cameras``)
* Isaac-Cartpole-v0
* Isaac-Ant-Direct-v0
* Isaac-Ant-v0
* Isaac-Humanoid-Direct-v0
* Isaac-Humanoid-v0
* Isaac-Velocity-Flat-Anymal-B-v0
* Isaac-Velocity-Flat-Anymal-C-v0
* Isaac-Velocity-Flat-Anymal-D-v0
* Isaac-Velocity-Flat-Cassie-v0
* Isaac-Velocity-Flat-G1-v0
* Isaac-Velocity-Flat-G1-v1 (Sim-to-Real tested)
* Isaac-Velocity-Flat-H1-v0
* Isaac-Velocity-Flat-Unitree-A1-v0
* Isaac-Velocity-Flat-Unitree-Go1-v0
* Isaac-Velocity-Flat-Unitree-Go2-v0
* Isaac-Velocity-Flat-Spot-v0
* Isaac-Reach-Franka-v0
* Isaac-Reach-UR10-v0
* Isaac-Repose-Cube-Allegro-Direct-v0


To launch an environment and check that it loads as expected, we can start by trying it out with zero actions sent to its actuators.
This can be done as follows, where ``TASK_NAME`` is the name of the task you’d like to run, and ``NUM_ENVS`` is the number of instances of the task that you’d like to create.
Expand Down
36 changes: 32 additions & 4 deletions docs/source/features/population_based_training.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,8 @@ You must start **one process per policy** and point them to the **same workspace
Minimal flags you need:

* ``agent.pbt.enabled=True``
* ``agent.pbt.workspace=<path/to/shared_folder>``
* ``agent.pbt.directory=<path/to/shared_folder>``
* ``agent.pbt.policy_idx=<0..num_policies-1>``
* ``agent.pbt.num_policies=<N>``

.. note::
All processes must use the same ``agent.pbt.workspace`` so they can see each other's checkpoints.
Expand All @@ -93,8 +92,37 @@ Minimal flags you need:
Tips
----

* Keep checkpoints fast: reduce ``interval_steps`` only if you really need tighter PBT cadence.
* It is recommended to run 6+ workers to see benefit of pbt
* Keep checkpoints reasonable: reduce ``interval_steps`` only if you really need tighter PBT cadence.
* Use larger ``threshold_std`` and ``threshold_abs`` for greater population diversity.
* It is recommended to run 6+ workers to see benefit of pbt.


Training Example
----------------

We provide a reference PPO config here for task:
`Isaac-Dexsuite-Kuka-Allegro-Lift-v0 <https://github.com/isaac-sim/IsaacLab/blob/main/source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/dexsuite/config/kuka_allegro/agents/rl_games_ppo_cfg.yaml>`_.
For the best logging experience, we recommend using wandb for the logging in the script.

Launch *N* workers, where *n* indicates each worker index:

.. code-block:: bash

# Run this once per worker (n = 0..N-1), all pointing to the same directory/workspace
./isaaclab.sh -p scripts/reinforcement_learning/rl_games/train.py \
--seed=<n> \
--task=Isaac-Dexsuite-Kuka-Allegro-Lift-v0 \
--num_envs=8192 \
--headless \
--track \
--wandb-name=idx<n> \
--wandb-entity=<**entity**> \
--wandb-project-name=<**project**>
agent.pbt.enabled=True \
agent.pbt.num_policies=<N> \
agent.pbt.policy_idx=<n> \
agent.pbt.workspace=<**pbt_workspace_name**> \
agent.pbt.directory=<**/path/to/shared_folder**> \


References
Expand Down
6 changes: 5 additions & 1 deletion docs/source/how-to/cloudxr_teleoperation.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -409,7 +409,7 @@ Manus + Vive Hand Tracking

Manus gloves and HTC Vive trackers can provide hand tracking when optical hand tracking from a headset is occluded.
This setup expects Manus gloves with a Manus SDK license and Vive trackers attached to the gloves.
Requires Isaac Sim >=5.1.
Requires Isaac Sim 5.1 or later.

Run the teleoperation example with Manus + Vive tracking:

Expand All @@ -425,6 +425,10 @@ Begin the session with your palms facing up.
This is necessary for calibrating Vive tracker poses using Apple Vision Pro wrist poses from a few initial frames,
as the Vive trackers attached to the back of the hands occlude the optical hand tracking.

For optimal performance, position the lighthouse above the hands, tilted slightly downward.
One lighthouse is sufficient if both hands are visible.
Ensure the lighthouse remains stable; a stand is recommended to prevent wobbling.

.. note::

To avoid resource contention and crashes, ensure Manus and Vive devices are connected to different USB controllers/buses.
Expand Down
2 changes: 1 addition & 1 deletion docs/source/setup/installation/include/pip_python_virtual_env.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ If you wish to install Isaac Sim 4.5, please use modify the instructions accordi

.. tab-item:: Conda Environment

To install conda, please follow the instructions `here <https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html>__`.
To install conda, please follow the instructions `here <https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html>`__.
You can create the Isaac Lab environment using the following commands.

We recommend using `Miniconda <https://www.anaconda.com/docs/getting-started/miniconda/main/>`_,
Expand Down
Loading
Loading