Support mixed shared and isolated CPU allocation with CPU DRA

[SchSeba]

Summary

Support a mixed CPU allocation mode where a Pod can request:

• isolated CPUs through CPU DRA
• additional fractional CPU from the regular shared CPU pool
  A concrete example is:
• Pod CPU request/limit: 5.5
• CPU DRA request: 5
  Expected result:
• 5 isolated CPUs are allocated through CPU DRA
• the remaining 0.5 CPU stays available for non-critical application threads

Motivation

High-performance workloads such as DPDK applications often need a fixed set of isolated CPUs for dataplane threads, but still need a small amount of non-dedicated CPU for helper, control-plane, logging, or housekeeping threads.
Today, the common workaround is to over-request dedicated CPUs and manage thread placement manually. That wastes isolated capacity and makes the deployment harder to express and automate.

Requested behavior

When a Pod combines a fractional CPU request with an integer CPU DRA request, the implementation should:

• allocate the integer DRA portion as isolated CPUs
• keep the fractional remainder outside the dedicated CPU set
• avoid forcing all workload threads onto the isolated CPUs
• preserve the dedicated CPUs for the latency-sensitive part of the workload
  Using the example above, a Pod that requests 5.5 CPUs and 5 CPUs through CPU DRA should end up with:
• 5 isolated CPUs for the dataplane / dedicated threads
• 0.5 CPU worth of regular shared runtime for non-critical threads

Application-facing API / discoverability

The implementation should also expose the allocated isolated CPU set to the Pod using the existing DRA application-facing mechanism, so user space applications can discover which CPUs are dedicated without relying on out-of-band configuration.
That data should be usable by applications such as DPDK to:

• pin dataplane threads to the dedicated CPU set
• keep helper / control threads off the dedicated CPUs
• document a recommended way to launch an application so non-critical threads do not consume the isolated cores

Why this matters

This would make it possible to express a very common real-world pattern:

• dedicated CPUs for the hot path
• a small shared CPU budget for everything else
  That improves utilization of isolated CPUs and reduces the need for manual cpuset handling in the application.

Acceptance criteria

• a Pod requesting 5.5 CPUs and 5 CPUs through CPU DRA gets exactly 5 isolated CPUs plus 0.5 CPU from the shared pool
• the isolated CPU set is exposed to the application through the existing DRA application-facing path
• example documentation shows how to pin dataplane threads to the dedicated CPU set and keep non-critical threads off those CPUs
• the behavior is clearly defined for NUMA alignment, SMT siblings, and interaction with CPU manager policies

[AutuSnow]

Thank you for opening this issue! This is a common real-world pattern for DPDK and latency sensitive workloads.
Currently, drivers operate in two modes:
-Containers with DRA declarations can only obtain isolated CPUs
-Containers without DRA declaration can only obtain shared pool CPUs
The implementation of this feature has raised many issues and challenges that need to be discussed

1. API design
   How should users express the intention of "5 isolated+0.5 shared CPUs"?
2. Container group configuration
   A practical question: Should a container have one or two containers?
   -Single CPU method: Set cpuset.CPU to all allowed CPUs (isolated+shared). The application needs to know which CPUs are isolated and fix threads accordingly. This is similar to how the kubelet CPU manager works today.
   -Dual CPU method: requires container level cgroup v2 tuning. The 'isolated' CPU will be located in cpuset.CPUs.effective, while the 'shared' CPU will be located in different cgroup controllers. This is more complex, but provides better isolation assurance.
3. Mechanisms for DRA applications
   The driver has exposed the allocated CPU set through the DRA_CPUSET_environment variable. Should we add additional metadata to distinguish between "isolation" and "sharing" in this situation, or is the current mechanism sufficient?
4. Backward compatibility, how should this feature recognize behavior?
   -Automatic: Whenever a Pod has both a fractional CPU request and a DRA declaration
   -Choose to join: Annotate through Pod

[fmuyassarov]

We actually discussed something very similar on the SIG Node mailing list last week. The question there was about enabling per-pod CPU frequency control (P-state knobs), so that low-priority pods could run at reduced CPU speeds without impacting other workloads on the same node. This feels closely related. The specific control is different, but the underlying scenario is similar, giving admins more granular control over how CPU resources are allocated between critical and non-critical workloads. In this case, it’s more about controlling which types of cores are assigned to critical workloads and allowing non-isolated CPUs to be used by sidecar or background pods.
I’m not sure if it makes sense to treat these as the same problem, but to me the mechanics seem quite similar - the main difference is in the knobs being exposed for each use case.

@ffromani, @pravk03 any thoughts?

[AutuSnow]

This lssue itself is a sig node email issue, and this question is only inquiring about the user's thoughts and expected goals, which has not reached the implementation stage

[fmuyassarov]

I agree this isn’t about implementation yet. My point was more about the similarity in the problem space, which could still be useful while we are defining expectations.

[AutuSnow]

Based on the two lssue questions raised by the user, it can be inferred that the current user should be the one who raised the question in the sig email. Of course, this is just my guess

[pravk03]

Some early thoughts on how we could approach this. Definitely needs more consideration.

How should users express the intention of "5 isolated+0.5 shared CPUs"?

With KEP-5517, the scheduler sums up requests in the container spec and claims for accounting. The most intuitive UX maybe to treat whatever goes in the container spec as the shared CPU request, and whatever is in the claim as the guaranteed/isolated CPU request. Until KEP-5517 graduates, we have to rely on comparing container spec request and claim request; container spec request > claim request could be considered as the intent for container requesting shared CPUs.

-Single CPU method: Set cpuset.CPU to all allowed CPUs (isolated+shared). The application needs to know which CPUs are isolated and fix threads accordingly. This is similar to how the kubelet CPU manager works today.

Yes, this seems like the easier of the two options. We could probably have a configurable flag to enable this mode. But the obvious drawback is that on the same node, we cannot have containers that want strict isolation and others that want mixed (isolated + shared). @SchSeba - Curious to know if this would work for your use case.
However, regarding the Kubelet comparison — I don't think Kubelet actually does this today. If the static CPU policy is enabled, if static CPU policy is enabled, a container either gets exclusive CPUs (if guaranteed QOS + integer requests) or shared pool.

Alternative - Infer intent for shared CPUs in NRI

We could infer the intent using the container sandbox's linux.resources.cpu.shares(set by kubelet). From the DRA_CPUSET env var, we know exactly how many isolated CPUs the claim allocated.

• If cpus calculated from shares equals the claim allocated CPUs - the user wants strict isolation. We only grant the isolated CPUs to the cpuset.
• If cpus calculated from shares > claim allocated CPUs (e.g., request is 5.5, claim is 5), we grant isolated + shared CPUs to the cpuset.

I will concede that this approach maybe hacky and brittle. I am not sure if there is a clean alternative to access the pod spec directly in the NRI.
The logic above maybe manageable for 1:1 container-to-claim mappings, but does not work if we consider shared claims across containers (which is currently disabled).

These are just my early thoughts. I think solving this safely adds a lot of complexity to the driver's state management. We'll definitely more discussions and prototyping if we want to solve this.

[ffromani]

Perhaps overlapping with #10

[ffromani]

Still catching up with the convo. Wanna share some early thoughts myself, and I'm only replying to @pravk03 because he happened to touch most of the topics I wanted to comment about :)

Some early thoughts on how we could approach this. Definitely needs more consideration.

How should users express the intention of "5 isolated+0.5 shared CPUs"?

With KEP-5517, the scheduler sums up requests in the container spec and claims for accounting. The most intuitive UX maybe to treat whatever goes in the container spec as the shared CPU request, and whatever is in the claim as the guaranteed/isolated CPU request. Until KEP-5517 graduates, we have to rely on comparing container spec request and claim request; container spec request > claim request could be considered as the intent for container requesting shared CPUs.

Indeed KEP-5517 is a major enabler for this usecase. We should also check how we can compose with pod-level resources and their resource manager enablement, which took a very similar use case explicitly into account.

-Single CPU method: Set cpuset.CPU to all allowed CPUs (isolated+shared). The application needs to know which CPUs are isolated and fix threads accordingly. This is similar to how the kubelet CPU manager works today.

Yes, this seems like the easier of the two options. We could probably have a configurable flag to enable this mode. But the obvious drawback is that on the same node, we cannot have containers that want strict isolation and others that want mixed (isolated + shared). @SchSeba - Curious to know if this would work for your use case. However, regarding the Kubelet comparison — I don't think Kubelet actually does this today. If the static CPU policy is enabled, if static CPU policy is enabled, a container either gets exclusive CPUs (if guaranteed QOS + integer requests) or shared pool.

Correct. While I didn't do an exhaustive research on the matter, I'm skeptical we can achieve full mixed CPU assignment (some isolated, some shared) to workload without some changes to the cgroups layout.
The problem is not making changes per se, it's likely (if memory serves) we would need limited changes and not a total overhaul. The problem is rather than kubelet owns the cgroup structure AND that structure is not standard but rather standardized as "whatever kubelet did, and now everyone depends on it and we can't change". So the only/easiest way out would be make kubelet able to defer the cgroup setup and layout and full ownership to someone else, be that someone a plugin, the runtime, the CRI standard...

Alternative - Infer intent for shared CPUs in NRI

We could infer the intent using the container sandbox's linux.resources.cpu.shares(set by kubelet). From the DRA_CPUSET env var, we know exactly how many isolated CPUs the claim allocated.

* If cpus calculated from shares equals the claim allocated CPUs -  the user wants strict isolation. We only grant the isolated CPUs to the cpuset.

* If cpus calculated from shares > claim allocated CPUs (e.g., request is 5.5, claim is 5), we grant isolated + shared CPUs to the cpuset.

I will concede that this approach maybe hacky and brittle. I am not sure if there is a clean alternative to access the pod spec directly in the NRI. The logic above maybe manageable for 1:1 container-to-claim mappings, but does not work if we consider shared claims across containers (which is currently disabled).

OTOH this is also a way to implement it, but I'd rather try hard (and document) to design the cleanest approach, map the gaps and get feedback from the sig-node community about appetite for it and timelines. And if the cleanest approach is nacked, iterate and accept compromises as needed.

These are just my early thoughts. I think solving this safely adds a lot of complexity to the driver's state management. We'll definitely more discussions and prototyping if we want to solve this.

True, but I'm quite sure we would need also quite some changes at kubelet and possibly CRI level to produce a robust solution.

[SchSeba]

Hi @pravk03 @ffromani,

I agree this is a complex topic.
I will need to check with the application owners if it's possible split base on containers and this way using the pod level resource + CPU DRA that should work right?

Another option I was thinking let me know if that is a good idea or make stuff more complicated is to have also a shared CPU as another device and this way the allocation is fully done via DRA where the user will request two different types of devices when it require both isolated and shared CPUs.
we pass the information into the container and the user needs to run the applications with the different CPUs

[pravk03]

I will need to check with the application owners if it's possible split base on containers and this way using the pod level resource + CPU DRA that should work right?

If we can have multiple containers, using claims for exclusive access in high priority containers while setting a pod-level budget (inclusive of claims) for other best effort containers in the pod should work well.

have also a shared CPU as another device and this way the allocation is fully done via DRA where the user will request two different types of devices when it require both isolated and shared CPUs.

In theory it must be possible to have a claim for shared CPUs reference it from all the containers needing them, but currently do not allow sharing claims between containers/pods. This was mostly to simplify the initial implementation, but I believe this can be revisited based on use-case. Sharing claims across pods might be more challenging for cgroup enforcement, but sharing claims between containers of the same pod should be feasible, especially if we leverage PLR to set pod level budgets.

[ffromani]

An important caveat is that IIRC we can' set quotas in the shared portion. All the container sharing a claim will have unbounded access to the same resource pool.

[pravk03]

An important caveat is that IIRC we can' set quotas in the shared portion. All the container sharing a claim will have unbounded access to the same resource pool.

Good point. Agree (I think you mean "can't" set quotas here).