Fairly share execution resources between users

[jonc125]

Celery is a handy task queue, but doesn't provide an easy mechanism for per-user rate limiting. This can only (easily) be set per-task. So if one user submits 50 experiments (say) they can monopolise all the workers until these are completed.

With WL1 we got around this by reserving some workers for admin users, and having a user->queue mapping so each user only had access to 2 workers. But this required updating the mapping manually for each user. It also means that in our current 'normal usage' case where only ~1 person is using the system at a time (unlike during a workshop!) most of the workers are idle.

Possible options to investigate:

1. Automate the creation of queues and assigning workers to them.
2. Have a 'celery beat' regular task that checks the state of the queue and re-orders tasks (by cancelling & re-submitting) to share fairly between users.
3. Users can spin up their own workers on Azure etc. and connect them to our broker. This would probably require some of (1) to manage the user->worker mapping too.