Hi Didier, I started two simultaneous runs: one that used the replay buffer and one that did not. I noticed that initial learning moved a lot quicker to higher scores when using ACER, but it eventually started flattening and slowing down, and was overtaken by the normal on-policy run. I just wanted to ask you if you knew of any hyperparameters or tweaks to the code from your experience that might change this behavior and make ACER converge quicker than normal on-policy.
Hi Didier, I started two simultaneous runs: one that used the replay buffer and one that did not. I noticed that initial learning moved a lot quicker to higher scores when using ACER, but it eventually started flattening and slowing down, and was overtaken by the normal on-policy run. I just wanted to ask you if you knew of any hyperparameters or tweaks to the code from your experience that might change this behavior and make ACER converge quicker than normal on-policy.