Potential issue and solution for poor high-level performance

[Ericonaldo]

Hi all,

Recently, @tenhearts reported a problem in high-level training: the object sometimes may penetrate the table, below is a screenshot from her. This may cause many failed grasp cases and she said that she finally got a great performance after fixing this problem (but maybe not the right solution).

At first I thought it was a bug of setting table heights, so I checked for a while. Finally, I found that it is not relevant to the code, but the problem of collision check of issacgym. It is important to note that the precision of collision check may related to the hardware and even the number of parallel environments, so I guess that may be the reason why people get different results on different machines. But it is stable on one particular machine and easy to reproduce.

According to the official document, we can change the contact_offset parameter (in config file) to prevent from similar problems. Special thanks to @CreeperLin to help me debugging on this.

Here are some examples I tested on my 4060 PC:

contact_offset=0.02, the default case in this codebase, very obvious penetration problem, the related environment is kept resetting due to the 'fall down' termination condition:

contect_offset_0.02-2024-08-08_19.21.58.mp4

changing contact_offset=0.04, the problem is well-solved.

contect_offset_0.04-2024-08-08_19.26.03.mp4

Can you guys try to train your own high-level model by fixing this issue? I'd like to help if there are more problems.

[hatimwen]

Thanks! I'll try it and share my results here later.

[sinaqahremani]

I have started the training considering this issue, I'll let you know the result tomorrow.

[sinaqahremani]

Prior to this idea executed the algorithm on a HP PC with i9 cpu, 32 GB Memory and RTX 6000 GPU and my fixed seed was 101. Now, I executed on the same system with contact_offset=0.04 and contact_offset=0.03 on the same system with the same seed. here are the success rates:

As you can see for my case the algorithm is getting worse.

[hatimwen]

Hi,

I've encountered an issue with the updated code when setting contact_offset=0.04. After approximately 20k training steps, the following errors consistently occur:

/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (1466) : internal error : Contact buffer overflow detected, please increase its size in the scene desc!


/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNphaseImplementationContext.cpp (710) : internal error : Contact buffer overflow detected, please increase its size in the scene desc!


/buildAgent/work/45f70df4210b2e3e/source/physx/src/NpScene.cpp (3189) : internal error : PhysX Internal CUDA error. Simulation can not continue!

/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (9908) : internal error : GPU compressContactStage1 fail to launch kernel stage 1!!


/buildAgent/work/45f70df4210b2e3e/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (9945) : internal error : GPU compressContactStage2 fail to launch kernel stage 1!!


[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 4084
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 4092
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3362
[Error] [carb.gym.plugin] Gym cuda error: an illegal memory access was encountered: ../../../source/plugins/carb/gym/impl/Gym/GymPhysX.cpp: 3417

Do you have any suggestions on how to resolve this?

Additionally, I ran some performance comparisons between contact_offset settings of 0.02 and 0.04. You can find the details here: wandb log. Setting contact_offset to 0.04 does show some improvements, but the issue mentioned above is a blocker.

Thanks in advance for your help!

[Ericonaldo]

Can you try to increase max_gpu_contact_pairs？

[hatimwen]

Can you try to increase max_gpu_contact_pairs？

ok, I will try it.

[sinaqahremani]

Any thoughts on my case? @Ericonaldo

[Ericonaldo]

@sinaqahremani Can you try to add a --debug para during training for visualizing and find some potential problems?

[sinaqahremani]

@Ericonaldo When I use --debug it won't upload data to Wandb!

[Ericonaldo]

you do not have to upload, just to visualize and check if there is anything weird

[sinaqahremani]

@Ericonaldo I executed in debug mode, nothing weird happened. however, the success rates are about zero, and they don't change.