Update to New RSL-RL - Enable multi gpu #31

alessandroassirelli98 · 2025-05-14T10:39:34Z

Update to New RSL-RL - Enable multi-gpu

Description

This pull request introduces significant improvements to the project by fully integrating the latest configuration of the RSL-RL library. The update removes the need for a custom version, enabling a more streamlined and modular workflow.

Summary of Changes

🆕 New Files

Added scripts/rsl_rl/cli_args.py and scripts/rsl_rl/student_policy_cfg.py to support the new configuration workflow.

✏️ Modified Files

README.md: Updated to reflect the latest usage instructions and improvements.
neural_wbc/: Modified multiple components, including environment wrappers, inference environments, and policy trainers, to align with the updated RSL-RL interface.
scripts/rsl_rl/: Revised training and evaluation scripts to support the latest features of the library.
pyproject.toml & requirements.txt: Updated dependencies to match the new setup.

🗑️ Removed Files

Deleted obsolete files under third_party/rsl_rl/, as the project now uses the upstream RSL-RL library directly.

Purpose

The main goals of this update are:

To adopt the upstream RSL-RL library, reducing maintenance overhead.
To simplify configuration and usage by removing the need to manually specify directories and checkpoints.
To enable multi-GPU training support through the latest RSL-RL features.

Testing

✅ Verified: Training, playing, and evaluation for both teacher and student policies.
⚠️ Pending: Sim2sim and sim2real deployment updates.
⚠️ Note: These changes are not backward compatible with the previous setup.

Let me know if there are any concerns or suggestions for further improvement.

fix missing attribute and log path add player for new rsl_rl remove checkpoint abspath. Updated doc running formatter adding student add check for multi gpu and set right device delete dependency remove install_deps.sh file remove custom rsl_rl fix deprecated attribute save log in robot subfolder configured training for student with new rsl_rl change observations teacher_policy to teacher for rsl_rl compatibility get right checkpoints in the player update code for eval run formatter fix eval script. updated doc add my isaaclab path to intellisense specify python package

pulkitg01 · 2025-05-22T20:00:46Z

Thanks for the MR! We really appreciate the contribution. It might take us a bit of time to review the changes thoroughly and test them out before we can merge it in. Thanks for your patience!

huihuaNvidia2023

First path done, need to verify locally.

huihuaNvidia2023 · 2025-05-23T21:42:55Z

.vscode/settings.json

This is nice but assumes a path of IsaacLab. A better way would be using the environmental variable ISAACLAB_PATH. VSCODE does provide utils to use env variable.

Yeah I was using this for debugging, I think it would be best to just remove it

README.md

huihuaNvidia2023 · 2025-05-23T21:48:24Z

neural_wbc/data/data/motions/stable_punch.pkl

We can't host this dataset due to licence issue.

scripts/rsl_rl/cli_args.py

huihuaNvidia2023 · 2025-05-23T22:03:23Z

scripts/rsl_rl/teacher_policy_cfg.py

-    num_steps_per_env: int = field(
-        default=24, metadata={"description": "Number of steps per environment per iteration."}
+    num_steps_per_env = 24
+    max_iterations = 10000000


nit: this number is too big. Normally 50k or 80k can give a good result, maybe 100k is enough?

Thanks a lot for the comprehensive changes, your contribution is much appreciated!

Actually, the latest IsaacLab 2.1 comes with the version of RSL RL that has distillation and a bunch other new features.

If we upgrade to v2.1.0, we could potentially make it even nicer. To jump from v2.0.0 to v2.1.0, the changes are minimum, only a few data names need to be updated.

It may worth to give it a shot.

For this I kept the same iteration number as the original version, but I can reduce it no problem.

Yeah so this PR already uses the version of RL_RL that has distillation. The only reason why I did not use it and I was installing rsl separately is because in the distillation algorithm rsl was not using a gradient cap, which was instead used in the original implementation shipped with HOVER. Now rsl has it, but as far as i know these changes have not been reflected in isaaclab which still uses rsl-rl-lib 2.3.1.

But this pr was mostly to include the new library so that we can use the multi gpu support. I know it's not super clean, but before spending too much time on it I wanted to know if you have any advice on how to proceed.

I'm very open to suggestions and I can work on it, so if you have ideas please share✌️

Thanks for mentioning the detail. Yeah, this works for me. Your PR is still the main piece. We can worry about the Lab conversion later.

huihuaNvidia2023 · 2025-05-27T17:16:53Z

scripts/rsl_rl/teacher_policy_cfg.py

-    num_steps_per_env: int = field(
-        default=24, metadata={"description": "Number of steps per environment per iteration."}
+    num_steps_per_env = 24
+    max_iterations = 10000000


Thanks for mentioning the detail. Yeah, this works for me. Your PR is still the main piece. We can worry about the Lab conversion later.

README.md

huihuaNvidia2023 · 2025-05-27T17:18:15Z

README.md

+It is possible to train on a node with multiple gpus or multiple nodes with multiple gpues.
+In order to train the teacher on a machine with 4 gpus:
+```bash
+python -m torch.distributed.run --nnodes=1 --nproc_per_node=4 scripts/rsl_rl/train_teacher.py --num_envs=1024 --headless --distributed


train_teacher.py -> train_teacher_policy.py

huihuaNvidia2023 · 2025-05-27T20:18:16Z

neural_wbc/isaac_lab_wrapper/tests/test_observations.py

can you please also update to use joint_friction_coeff here

On another note, there are several unit tests are failing in the core, inference_env and isaac_lab_wrapper folders. Could you please take a look?

huihuaNvidia2023 · 2025-05-27T20:27:04Z

scripts/rsl_rl/student_policy_cfg.py

+    # Add max_grad_norm to the configuration
+    # As it has not been added to isaaclab yet
+    max_grad_norm: float = MISSING
+    pass


nit: remove

alessandroassirelli98 · 2025-08-20T09:15:07Z

Hey there!
Sorry for the long absence, but I have been very busy for a while.
I just merged the latest changes from the main branch. please let me know how you want to proceed with this.

I checked the tests.
in core I get a failure also with the main branch, due to the fact that the returned value from ReferenceMotionManager is a tensor and not a float. So i updated the test. However it still fails because it is comparing two quaternions that are different, I think this depends on the loaded motion though.
The rest is now passing.

alessandro.assirelli added 2 commits May 14, 2025 12:01

add doc for multi gpu

232c608

yanchangNvidia requested a review from huihuaNvidia2023 May 22, 2025 21:17

huihuaNvidia2023 reviewed May 23, 2025

View reviewed changes

update readme

9992b91

huihuaNvidia2023 reviewed May 27, 2025

View reviewed changes

huihuaNvidia2023 mentioned this pull request May 27, 2025

Add student onnx export #26

Open

alessandro.assirelli added 4 commits August 20, 2025 06:46

Merge remote-tracking branch 'mygithub/main' into squash-new-rsl_rl

bd8a497

implement abstract method

f91d8bd

update test

2f74665

fix unittest

797dc3f

Update to New RSL-RL - Enable multi gpu #31

Are you sure you want to change the base?

Update to New RSL-RL - Enable multi gpu #31

Uh oh!

Conversation

alessandroassirelli98 commented May 14, 2025

Update to New RSL-RL - Enable multi-gpu

Description

Summary of Changes

🆕 New Files

✏️ Modified Files

🗑️ Removed Files

Purpose

Testing

Uh oh!

pulkitg01 commented May 22, 2025

Uh oh!

huihuaNvidia2023 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alessandroassirelli98 commented Aug 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants