DRA: Handle extended resource requests via DRA Driver

[klueska]

Enhancement Description

• One-line enhancement description (can be used as a release note):
  Allow DRA drivers to honor requests made via the extended resource API (e.g. nvidia.com/gpu: 2) rather than requiring a standard device plugin be used.

• Kubernetes Enhancement Proposal:

  • https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/5004-dra-extended-resource

• Discussion Link:

  • https://youtu.be/fKhX_lHK8Z0?si=gq5kIFHP3ve2TXyE&t=1822

• Primary contact (assignee):
  @klueska, @pohly, @johnbelamaric

• Responsible SIGs:
  /sig node
  /wg device-management

• Enhancement target (which target equals to which milestone):

  • Alpha release target: 1.34
  • Beta release target: 1.35
  • Stable release target: 1.36

• Alpha (1.34)

  • KEP (k/enhancements) update PR(s):
    • Add KEP for DRA: Extended Resource #5136
  • Code (k/k) update PR(s):
    • kubelet and scheduler for extended resource backed by DRA kubernetes#130653
  • Docs (k/website) update PR(s):
    • [KEP-5004] DRA Extended resource documentation website#51710
  • Known issues where fixes won't be backported:
    • resource claims created by scheduler for extended resources are not de-allocated when pod is completed kubernetes#134311

• Alpha

  • KEP (k/enhancements) update PR(s):
    • KEP-5004: DRA Extended Resource: graduate to beta #5537
    • changed 1.36 to beta for DRA extended resource feature #5702
  • Code (k/k) update PR(s):
    • added scorer: DRA extended resources scorer kubernetes#133669
    • added resource quota: DRA extended resources resourcequota kubernetes#133671
    • support multiple device class for the same extended resource: Pick predicatable device class when multiple device classes use the same extended resource name kubernetes#133693
    • added metric: DRA extended resources monitoring requirements kubernetes#133758
  • Docs (k/website) update(s):
    • [KEP-5004] DRA Extended resource quota documentation website#52850

• Beta (1.36 - see DRA Extended Resource: Promotion to Beta kubernetes#133675)

  • KEP (k/enhancements) update PR(s):
    • KEP-5004: DRA Extended Resources: promotion to beta #5785
    • KEP-5004: Add test planning for Rollout, Upgrade and Rollback scenarios #5751
  • Code (k/k) update PR(s):
    • Fixes the loophole that allows users to workaround resource quota set by system admin kubernetes#135434
    • Fix extended resource handling for DRA-backed resources on pod admission kubernetes#135725
    • DRA Extended Resource: promote to Beta in 1.36 kubernetes#135048
  • Docs (k/website) update(s):
    • promote DRA extended resource feature to beta. website#54652

[johnbelamaric]

+1 yes please!

[johnbelamaric]

We need to sort out the requirements. A few initial questions:

1. For newly created pods, I think it's clear we want this to be transparent. Existing manifests that use the extended resource API should continue to work as before, without modification.
2. Can we handle this invisibly in the driver layer, or do we need to have DRA invoked at the control plane level and select the specific devices? If we don't, we will likely have a race condition - unless the scheduler can do some magical accounting (which seems possible).
3. How do we handle upgrades? If we have a node running device plugin, and we switch to the DRA driver (or we upgrade to a driver that supports both), do you have to delete the pods? Do they automatically adopt the devices? If so, how do we write those back to the allocation logic (since no DRA claim exists).
4. What happens if there are pods in a deployment, and some land on nodes with device plugin and some with DRA drivers?
5. We talked about letting specific device classes be advertised as specific extended resources. This could mean the existing resource names get mapped to specific device classes by the admin. It could also mean we have a convention like deviceclass.k8s.io/foo: 4 for extended resource names. How do these choices interplay with the questions above?

[lengrongfu]

Can each dra-driver implement a webhook to create a ResourceClaimTemplate after creating a pod and modify the application method of resources in the pod?

[klueska]

@lengrongfu that is what this KEP would be designed to avoid. There would be integrated scheduler support for all drivers, rather than requiring each DRA driver to provide a webhook.

[alculquicondor]

Open questions (from SIG Scheduling meeting):

• How to handle resource quotas
• Scheduling throughput (API requests and overall processing).

[ffromani]

/cc

[yliaog]

/cc

[johnbelamaric]

/sig scheduling