NaN in actions when training elevation task

[HHADS-dev]

Hi, thanks for the great work.
I met an error when running the elevation task training using
python source/wheeledlab_rl/scripts/train_rl.py --headless -r RSS_ELEV_CONFIG
However I meet the error

After training for 67 iteration
Has anyone meet this error before? Any suggestion for the potential cause? Thanks!

[geiko246]

Thanks for bringing this to our attention!

This would mean that either an observation is NaN or the network weights NaN'd in the previous iteration (which I think is unlikely).

We might have encountered this before - @10preet could you take a look as well?

Do you encounter this issue consistently? As a temporary workaround I would suggest:

1. adding a bit (~0.1) to the starting z-position, or
2. Clipping the remaining observation terms that aren't already clipped here:

   WheeledLab/source/wheeledlab_tasks/wheeledlab_tasks/elevation/mushr_elevation_env_cfg.py

   Lines 61 to 82 in 09594e5

   	class ConcatObs(ObsGroup):
   	goal_relative_xyz = ObsTerm(
   	func=goal_relative_xyz,
   	)
   	world_euler_xyz = ObsTerm(
   	func=root_euler_xyz,
   	)
   	base_lin_vel = ObsTerm(func=mdp.base_lin_vel, clip=(-10., 10.))
   	base_ang_vel = ObsTerm(func=mdp.base_ang_vel, clip=(-10., 10.))
   	last_action = ObsTerm(
   	func=mdp.last_action,
   	clip=(-1., 1.)
   	)
   	elevation_map = ObsTerm(
   	func=world_height_map,
   	params={
   	"sensor_cfg":SceneEntityCfg("height_scanner"),
   	"offset": 0.084,
   	"plane_init_value": 0.19
   	},
   	clip=(-10., 10.),
   	)

   
   I don't believe the value to which you're clipping matters too much as long as it's finite.

[HHADS-dev]

Thanks for bringing this to our attention!

This would mean that either an observation is NaN or the network weights NaN'd in the previous iteration (which I think is unlikely).

We might have encountered this before - @10preet could you take a look as well?

Do you encounter this issue consistently? As a temporary workaround I would suggest:

1. adding a bit (~0.1) to the starting z-position, or

2. Clipping the remaining observation terms that aren't already clipped here: 
     
       
         [WheeledLab/source/wheeledlab_tasks/wheeledlab_tasks/elevation/mushr_elevation_env_cfg.py](https://github.com/UWRobotLearning/WheeledLab/blob/09594e5487462f22c5266ad3472ab8bd0cdec927/source/wheeledlab_tasks/wheeledlab_tasks/elevation/mushr_elevation_env_cfg.py#L61-L82)
       
       
           Lines 61 to 82
         in
         [09594e5](/UWRobotLearning/WheeledLab/commit/09594e5487462f22c5266ad3472ab8bd0cdec927)
       
     
     
       
   
           
             
              class ConcatObs(ObsGroup): 
           
   
           
             
                  goal_relative_xyz = ObsTerm( 
           
   
           
             
                      func=goal_relative_xyz, 
           
   
           
             
                  ) 
           
   
           
             
                  world_euler_xyz = ObsTerm( 
           
   
           
             
                      func=root_euler_xyz, 
           
   
           
             
                  ) 
           
   
           
             
                  base_lin_vel = ObsTerm(func=mdp.base_lin_vel, clip=(-10., 10.)) 
           
   
           
             
                  base_ang_vel = ObsTerm(func=mdp.base_ang_vel, clip=(-10., 10.)) 
           
   
           
             
                  last_action = ObsTerm( 
           
   
           
             
                      func=mdp.last_action, 
           
   
           
             
                      clip=(-1., 1.) 
           
   
           
             
                  ) 
           
   
           
             
                  elevation_map = ObsTerm( 
           
   
           
             
                      func=world_height_map, 
           
   
           
             
                      params={ 
           
   
           
             
                              "sensor_cfg":SceneEntityCfg("height_scanner"), 
           
   
           
             
                              "offset": 0.084, 
           
   
           
             
                              "plane_init_value": 0.19 
           
   
           
             
                          }, 
           
   
           
             
                      clip=(-10., 10.), 
           
   
           
             
                  ) 
           
       
     
   
   
   I don't believe the value to which you're clipping matters too much as long as it's finite.

Thanks for your reply. I rerun the training, the error also appear after same iteration. I will try your suggestion to see whether the error will be solved. Thanks!

[HHADS-dev]

Hi，I tried the solution you suggests, the same occurs in the later iteration. It shows that there is NaN in the obs and critic_obs. Are there any other potential cause for this error? And I also want to ask how long it will take for the elevation training? I run the training on a A6000, It shows that ETA is around 10 hrs. Is it normal? Thanks

[geiko246]

That does not seem right. It should only take a couple hours. I can try running this on my system and check as well.

Just to confirm - you clipped all of the observations and it still occurs? That's strange. I suppose that might indicate Isaac Sim itself is throwing NaNs.

[jazonshou]

@HHADS-dev Do you remember which iteration the error occurs (after implementing the workaround)?

[HHADS-dev]

@HHADS-dev Do you remember which iteration the error occurs (after implementing the workaround)?

Hi, sorry for the reply, the error occurs on the iteration 72 after implementing the workaround

[HHADS-dev]

That does not seem right. It should only take a couple hours. I can try running this on my system and check as well.

Just to confirm - you clipped all of the observations and it still occurs? That's strange. I suppose that might indicate Isaac Sim itself is throwing NaNs.

yes, the error still occur but in later iteration after clipping all the observation.

[Ezio1018]

                     �[1m Learning iteration 169/5000 �[0m                      

                       Computation: 8834 steps/s (collection: 14.744s, learning 0.092s)
             Mean action noise std: 0.95
          Mean value_function loss: 1542116.2875
               Mean surrogate loss: -0.0005
                 Mean entropy loss: 2.7411
                       Mean reward: 4761.34
               Mean episode length: 45.09
   Episode_Reward/vel_towards_goal: 187.1951
           Episode_Reward/height_z: 0.3630
    Episode_Reward/falling_penalty: 0.0217
Episode_Reward/termination_penalty: -0.8504
       Metrics/goal_pose/error_pos: 0.0000
   Metrics/goal_pose/error_heading: 0.9777
    Metrics/goal_pose/error_pos_2d: 19.5310
      Episode_Termination/time_out: 1.6875
Episode_Termination/cart_out_of_bounds: 1.2266
         Episode_Termination/stuck: 23.8672
      Episode_Termination/rollover: 1.6562
       Episode_Termination/at_goal: 0.1641
--------------------------------------------------------------------------------
                   Total timesteps: 22282240
                    Iteration time: 14.84s
                      Time elapsed: 00:48:31
                               ETA: 22:58:51

   3% ━                               170/5,000  [ 0:48:49 < 19:57:20 , 0 it/s ]
Error executing job with overrides: []
Traceback (most recent call last):
  File "/root/WheeledLab-main/source/wheeledlab_rl/wheeledlab_rl/utils/hydra.py", line 160, in hydra_main
    func(run_cfg, *args, **kwargs)
  File "/root/WheeledLab-main/source/wheeledlab_rl/scripts/train_rl.py", line 118, in main
    runner.learn(num_learning_iterations=train_cfg.num_iterations)
  File "/root/WheeledLab-main/source/wheeledlab_rl/wheeledlab_rl/utils/modified_rsl_rl_runner.py", line 75, in learn
    raise ValueError("NaN in actions")
ValueError: NaN in actions

The same problem occurs to me

[Ezio1018]

def goal_relative_xyz(env : ManagerBasedEnv):
pos = mdp.root_pos_w(env)
goal_pos = mdp.generated_commands(env, "goal_pose")
goal_pos = goal_pos[:, :2] # we only need the x, y coordinates
rel_pos = goal_pos - pos[:, :2]

if torch.isnan(rel_pos).any():
    nan_mask = torch.isnan(rel_pos)
    if nan_mask.any():
        nan_indices = nan_mask.any(dim=1).nonzero(as_tuple=True)[0]
        print("NaN detected in goal_relative_xyz")
        print("Indices with NaN:", nan_indices)
        for idx in nan_indices.tolist():
            print(f"\n--- env[{idx}] ---")
            print("goal_pos:", goal_pos[idx])
            print("robot_pos:", pos[idx])
            print("rel_pos:", rel_pos[idx])
        raise ValueError("NaN in goal_relative_xyz")

return torch.nan_to_num(rel_pos, nan=0)

output:

NaN detected in goal_relative_xyz
Indices with NaN: tensor([622], device='cuda:0')

--- env[622] ---
goal_pos: tensor([nan, nan], device='cuda:0')
robot_pos: tensor([nan, nan, nan], device='cuda:0')
rel_pos: tensor([nan, nan], device='cuda:0')

[geiko246]

That's a great find. I suppose that means the pose itself is NaN'd since goal relative should just be a finite operation. I'm really confused how that would be possible though.

I guess the workaround for now can be this nan_to_num observation term.

[Ezio1018]

That's a great find. I suppose that means the pose itself is NaN'd since goal relative should just be a finite operation. I'm really confused how that would be possible though.

I guess the workaround for now can be this nan_to_num observation term.

I suspect it was because the initial position of the car overlapped with obstacles in the map. I increased the height value in the post_init(), and no issues have been found for now.

[jazonshou]

I increased the height value in the post_init(), and no issues have been found for now.

That's pretty sweet. What height value did you increase it to? I've tried this before but the error still shows up. I think I might've not increased it enough.

[geiko246]

Do you mind sharing or opening a PR so we can try your fix/edits?

[Ezio1018]

Increasing the obstacle height only extended the training to 2000 steps and still didn’t solve the problem. I also tried using TerrainBasedPose2dCommandCfg and reset_root_state_from_terrain from IsaacLab to sample valid positions, but it seems doesn't support the usd terrain. This is all the information I can provide for now.