RFC: Admission controller proof of concept

[mismithhisler]

Description

DO NOT MERGE

As a part of some investigative work around Nomad fair scheduling in resource constrained clusters, we decided to take a look at what fair resource sharing would look like as an implementation of a generic admission controller. Our initial implementation of this admission controller pushed jobs to a queue and kept track of reserved cpu and memory at the namespace level (although we had plans for more configurable tenancy models). When resources for the configured node pool became constrained, and jobs begin to queue, a simple fairsharing algorithm would decide which one to run next (Using the SkipEvalCreation flag, a controller could mutate the job so it was stored in Raft but not run until the controller forces an evaluation).

Using a configuration like the below would configure the nomad server to forward job

server {
  admission_controllers {
    external "fairshare" {
      endpoint  = "http://127.0.0.1:8000/v1/jobs"
      node_pool = "some_nodepool"
    }
  }
}

Fairsharing algorithms are inherently stateful which complicated the controller enough that it did not seem to provide much more benefit than submitting jobs directly to a queue (which is what most users do with this use case).

We are currently investigating other places in Nomad where this type of logic might fit better, but wanted to push up these changes in the event others may have a good use case or ideas for external admission controllers.

Request for comments

Please share if you could use admission controllers and if you have feedback on this design. While we do not have admission controllers on the roadmap yet, we have long intended to make admission-controller-like patterns easier to implement.

Feedback welcome!