Relationship between number of samples and number of iterations

[meghbhalerao]

Hi Rutav,
The plots that are provided in the paper - example given below plot the number of samples vs the success rate

I also see that the code logs the success rate after every iteration as mentioned here - https://github.com/ShahRutav/mjrl/blob/6cdb8b8c72279abe8d9d8b8a800f8ac396413e42/mjrl/utils/train_agent.py#L119 and according to the default configuration file here - https://github.com/facebookresearch/RRL/blob/main/examples/config/hammer_dapg.yaml#L40 - the code is being run for 200 iterations, I also see here https://github.com/facebookresearch/RRL/blob/main/examples/config/hammer_dapg.yaml#L40 that the number of trajectories is 200, so I think the horizon length, say h, has to be 100, assuming that the training is happening for 4 x 10e6 if I am not wrong - I have the following doubts -

1. Here - https://github.com/ShahRutav/mjrl/blob/6cdb8b8c72279abe8d9d8b8a800f8ac396413e42/mjrl/algos/batch_reinforce.py#L64 I see that the horizon length is 1e6 - am I looking at the right place for that or is there some other parameters that I am missing?
2. Is the eval_success being logged at each iteration according to the code and the plots which are reported in the paper simply have their axis scaled by an appropriate factor, in our case something like 2 x 10eX (X depends on the answer to the above point) - so essentially the total number of logged eval_success is 200?

Please do let me know if my understanding is right in this setting.
Thanks,
Megh

[ShahRutav]

Hi,

1. That is just an upper bound, all the environments return done as True after n steps (the actual env horizon) where n is defined here

2. I am not sure if I understood your question correctly. The x-axis labels are in millions: 4M is 4 million.

[meghbhalerao]

Thanks - so when I run the code for resnet34 and log the success_rate after every iteration, whenever it is calculated in the script I get the following plot (I am making no changes to the hydra configuration file - this is for hammer task)

The above plot is for 1 seed. From what I understand by comparing the plot in the paper vs the plot I obtained it looks like I am running the training for more time?

Please let me know if I am missing something. Thanks for your help.

[ShahRutav]

what is your x-axis? I am not sure. Can you plot success rate Vs number of samples

[meghbhalerao]

so basically I am logging eval_success at this line https://github.com/ShahRutav/mjrl/blob/6cdb8b8c72279abe8d9d8b8a800f8ac396413e42/mjrl/utils/train_agent.py#L119 and hence we get 200 datapoints for eval_success each corresponding to one iteration. Is this the correct way to log?