Recommended approach to reliably test a OneShotAgent

[eranhirs]

After adding all the winners from last year and running myagent.py (from the skeleton project), we get pretty much random results, each time a different agent wins, even simple greedy ones.

We tried n_steps=50 and n_configs=5, should we try larger values?
What is the recommended approach / configuration to reliably test a OneShotAgent?

[yasserfarouk]

Hi @eranhirs

The differences between agents we had from last year are pretty small in reality. This is why we needed to run an extremely large tournament to get reliable results (1800 configurations if I remember correctly) and even after that the differences between the top 4 agents were small (that is why we had a tie in the third place). I do not recommend doing that as it will require huge computation though.

If you want to get more stable results, using a larger n_config will be more helpful than a larger n_step value.

I suggest using a tournament with at least 50 configs to get a stable result. Please note though that the ordering of agents you will get may differ from their ordering in our 2021's iteration of the competition. The reason is that the score of an agent depends on what other agents exist in the enviornment and you are not using non-winners which changes the enviornment.

Please also take a look at our controlled experiments tutorial which shows ways to test agents in specific conditions.