Opponent selection strategy during training?

[maxmax1992]

Hi,
I'm sorry to say but it's hard to follow the logic how the opponents are chosen for the PPO agent.

I'm aware of very basic strategy that basically past N models that are checkpointed are selected as an opponent (by random), for the main trained network. Source is from OpenAI

If you could elaborate how you did this? There's some kind of ELO ranking going on but I'm not completely sure if it's just for logging or it's used for opponent selection.

[AlperSekerci]

ELO ranking is used for opponent selection. Assume we have 4 agents checkpointed: 400, 300, 200 and 100 ELO. Our training agent observes 10,000 states during a single iteration (10k is a hyperparameter). 2,500 observations are from the matches against the 400 ELO opponent. 2,500 observations are against 300 ELO and so on. The training agent spends equal time with the strongest 4 opponents. Note that the number of matches against each agent may not be equal.

When the fixed amount of observations are collected, I calculate the win rates against each agent. Using these results, I can calculate the difference in ELO rating against each opponent. For instance, 64% win rate means 100 ELO difference (200 ELO = 76%, 300 ELO = 85%, 400 ELO = 91%). If all these 4 win rates are achieved, I freeze the training agent and call it 500 ELO. The required ELO difference for promotion is also a hyperparameter. You can use this site to calculate the expected win rate for a given ELO difference: https://wismuth.com/elo/calculator.html

[maxmax1992]

Thanks,

I think that explains a bit, but not completely. I guess I have to look more on the code to understand the whole process. Like after 10k steps do you also perform ELO score recalculation for all past models?

Also, what is the difference with your approach and just basic approach where you keep 4 latest models and train against them? Have you tried tried different approaches? Like selecting agent from past N models that is chosen Uniformly? Also there's something like this which something similar with WR recalculation.
https://arxiv.org/pdf/2009.06086.pdf