Suggestion: more control over testing setup

[jacksondc]

Use case: what we want to do is generate 10 or 20 world configurations, and test five different versions of our agent (with five different parameter settings) in each of those world configurations. (The tournament interface isn't ideal because we don't care if they play against each other). We hacked together a script to do this, which I think is working, but it was a bit of a pain.

What might make it easier:

1. Is there a way to set a random seed for a simulation, so that you can run exactly the same simulation multiple times? Or is there a way to write the parameter settings to a file, and then read them back in for repeated testing of the same parameter configuration?
2. More control over the tournament set-up—in particular it seems to generate very large numbers of worlds for 3+ competitors, such that the tournaments take too long to finish on our machines. It would be nice to be able to control the number of worlds generated.

[yasserfarouk]

Is there a way to set a random seed for a simulation, so that you can run exactly the same simulation multiple times?

Controlling randomness in python is painful. NegMAS and SCML use only the standard random generators of random and numpy.random modules. If you seed these two, runs should be consistent. One problem though is that other agents may use other libraries that have their own independent random generation (e.g. pytorch and tensorflow I think). That cannot be seeded except if you already know all of these libraries and seed them all at the start of the script.

Or is there a way to write the parameter settings to a file, and then read them back in for repeated testing of the same parameter configuration?

That depends on whehter or not you are using the tournament interface. If you are using the tournament interface, then you can just pass dry_run=True to the tournament function you are using (i.e.g anac2020_.....). This will generate the tournament and save all configs to a folder (the default location is under ~/negmas/tournaments.

After that you can use negmas.tournaments.run_tournament() to run the tournament as many times as you want.

More control over the tournament set-up—in particular it seems to generate very large numbers of worlds for 3+ competitors, such that the tournaments take too long to finish on our machines. It would be nice to be able to control the number of worlds generated.

The tournament generator tries to make sure all possibilities are tried. There are multiple ways to control that:

1. You can pass max_worlds_per_config which limits the number of simulations per configuration. The price you pay is that not all agents get to control all assignable factories in all simulations.
2. The explosion happens if you pass n_competitors_per_world > 2 because all permutations are generated. Limiting this to 2 makes two simulations per configuration.
3. There is only n_runs_per_world which controls how many times each specific world configuration with specific agent assignments is run. I think it is safe to pass this as 1 in most cases (specially if you are seeding randomness). The default is 5.
4. You can also try using round_robin=False if n_competitors_per_world divides len(competitors). This corresponds to running a loser-out kind of competition instead of a full tournament and will reduce the number of simulations a lot.

Finally, I agree that the tournament interface is not ideal for doing experiments. It was designed for tournaments. We actually have an unsupported experiments module in the SCML package which is designed for running experiments but it is far from being ready.

For now, I think it is better to directly generate workds using the generate() method of your world class instead of relying on the tournament interface if you want full control.

Another possibility is to use create_tournament, run_tournament and evaluate_tournament methods of the NegMAS tournaments module directly rather than using the anac_...() helpers. You will have to create your own world_config_generator, world_assigner and world_generator but you will have full control over all aspects of the process.

[jacksondc]

Awesome, we will take a look into all this!