Add priority to argo workflows

[KPostOffice]

Is your feature request related to a problem? Please describe.
User requests should always be processed first. We could add the priority field to all of the workflows to influence the order which workflows are processed in.

Describe the solution you'd like

ctrl+f "priority"
https://argoproj.github.io/argo-workflows/fields/

This issue is meant to be both a proposal for priority and a place to discuss potential priorities if we decide we would like to add them.

[goern]

/sig user-experience

[VannTen]

Would we also consider priority at the Template level ?
(This is different from pod priority for kube-scheduler right ? That would go via priorityClass, I think ?)

This is also relevant for devsecops, it could be used to have all jobs in one namespace and use one resource quota and priority (rather than multiples resource quota) for better resource utilization.

/sig devsecops

[KPostOffice]

Would we also consider priority at the Template level ? (This is different from pod priority for kube-scheduler right ? That would go via priorityClass, I think ?)

Yeah exactly. This priority is specifically for the Argo controller

This is also relevant for devsecops, it could be used to have all jobs in one namespace and use one resource quota and priority (rather than multiples resource quota) for better resource utilization.

This is an interesting point. The only thing that I would worry about is possible starvation of lower priority tasks, I'm not sure how Argo Workflows implements priority. (Partial DOS if many user requests come at once)

Argo Workflows also allows for semaphores which can be used to limit number of parallel workflows which all require the same lock. I'm unsure how these interact with priority.

[VannTen]

This is an interesting point. The only thing that I would worry about is possible starvation of lower priority tasks, I'm not sure how Argo Workflows implements priority. (Partial DOS if many user requests come at once)

Hum, this makes me thinks that we would also need to check if there is something akin to pod-preemption. Probably warrant a separate issue.

[harshad16]

Suggestions:

• Make adviser run have priority over kebechet runs

[harshad16]

/triage accepted
/priority important-longterm