Visual policy training very slow

[MuammerBay]

Hey,

just wanted to ask if it is normal that the visual policy training is running very slow. Even though the TiledCamera is usually slow, this one seems to be even slower as I got 13/5000 steps within 15minutes running on my RTX3090.

Perhaps this is due to recent changes in IsaacLab?

[geiko246]

Thanks for the issue! We saw this on our side when we migrated over to IsaacLab v2 to release the code. While we were still working on the paper, the visual policy would train in ~2 hours so the current progress rate is not expected.

I saw that there were some changes to the TiledCamera which might have affected things.

@shjung13 is currently working on addressing this.

[shjung13]

Hi MuammerBay,

We found that in our previous versions, we used 128 environments, while it is currently set to 512, which is four times greater than before.

Reducing the number of environments will help you in reproducing our performance in the paper and also reducing the training time.

[shjung13]

I will update this issue after investigating it on my side.