Respection native k8s preemption policy

[drubinstein]

What you would like to be added?

It is unclear if a custom priority class's preemption policy is currently respected by Kai.

Instead of the <100 heuristic, would it be possible to respect a priority class's preemption policy?

Why is this needed?

It will create less confusion when managing the Kai scheduler.

[enoodle]

@drubinstein The short answer is that I don't think KAI can / should fully support it.

On priority value alone:

KAI has a fairness system between its Queues and uses the same priority classes to sort jobs in each queue, but it is not used to change the order of the queues (meaning which queue will get to allocate the next job) .
So you can say that within a single KAI Queue this priority policy is kept - If there are no more resources then higher priority jobs will preempt lower priority jobs.

About the preemptionPolicy: Never that you linked directly - This can be implemented in KAI.