WIP: group CPUs by PCIE root

[ffromani]

Group CPUs by their PCIE root locality. We gain compatibility with all the kubernetes-compliant drivers, and we still allow optimal resource allocation

more context: kubernetes/kubernetes#132296 (comment)

WIP: needs tests, polishing, docs

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ffromani

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [ffromani]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ffromani]

this is getting weirder:

fromani@laptop:~/go/src/sigs.k8s.io/dra-driver-cpu$ pre-commit run --all-files
check for merge conflicts................................................Passed
check that executables have shebangs.....................................Passed
check that scripts with shebangs are executable..........................Passed
check json...........................................(no files to check)Skipped
check yaml...............................................................Passed
check for broken symlinks............................(no files to check)Skipped
check for added large files..............................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
detect private key.......................................................Passed
mdformat.................................................................Passed
codespell................................................................Passed
shfmt....................................................................Passed
ShellCheck v0.10.0.......................................................Passed
go-fmt...................................................................Passed
go-mod-tidy..............................................................Passed
go-vet-mod...............................................................Passed
go-build-mod.............................................................Passed
Make Tests...............................................................Passed
fromani@laptop:~/go/src/sigs.k8s.io/dra-driver-cpu$ make test-unit &> /dev/null 
fromani@laptop:~/go/src/sigs.k8s.io/dra-driver-cpu$ echo $?
0
fromani@laptop:~/go/src/sigs.k8s.io/dra-driver-cpu$ 

[ffromani]

CI woes addressed in #70

[ffromani]

Let's see how it could look like. I'm using this tool I wrote and I'm looking at a dual-Xeon Gold 6230R CPU which I have access to.
YAML used:

apiVersion: v1
kind: Pod
metadata:
  generateName: chk-pod-
spec:
  containers:
  - name: ctrreschk
    image: quay.io/fromani/ctrreschk:v0.0.11
    imagePullPolicy: Always
    command: ["/ctrreschk", "-w", "align"]
    resources:
      limits:
        cpu: 1
        memory: 256Mi
      requests:
        cpu: 1
        memory: 256Mi

output:

kubectl exec -ti  chk-pod-v4t2z -- /ctrreschk pciescan
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:00" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102" "NUMANode"=0
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:17" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102" "NUMANode"=0
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:3a" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102" "NUMANode"=0
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:85" "localCPUs"="1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103" "NUMANode"=1
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:ae" "localCPUs"="1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103" "NUMANode"=1
2026/03/17 11:30:50 "level"=0 "msg"="PCIE domain" "root"="pci0000:d7" "localCPUs"="1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103" "NUMANode"=1

we can see we detect 3 PCIE roots per NUMA node. The fact we have more than 1 PCIE root per NUMA node is not surprising: this fact is the very reason why we are moving to use PCIE root locality as better attribute to align resources.

The representational problem, however, is that we can have more than a PCIE root which takes resources from the same pool, be them CPUs or NUMA nodes. I'm not sure we can correctly represent the HW topology with consumable capacity alone. I tend to believe we would need kubernetes/enhancements#5942 to be available before we can make progress in this area.

[AutuSnow]

I'm thinking that we should treat PCIe Root not as a boundary of resource capacity, but as a local label, and standardize the calculation algorithm to become a consensus across drivers. This ensures the correctness of capacity representation (NUMA grouping, no overlap) and achieves PCIe alignment across drivers. However, this issue may require extensive discussion

[fmuyassarov]

we can see we detect 3 PCIE roots per NUMA node. The fact we have more than 1 PCIE root per NUMA node is not surprising: this fact is the very reason why we are moving to use PCIE root locality as better attribute to align resources.

The representational problem, however, is that we can have more than a PCIE root which takes resources from the same pool, be them CPUs or NUMA nodes. I'm not sure we can correctly represent the HW topology with consumable capacity alone. I tend to believe we would need kubernetes/enhancements#5942 to be available before we can make progress in this area.

Did a quick test as well and as you @ffromani already mentioned, a core is listed in more than one PCIe Root:

2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:00" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126" "NUMANode"=0
2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:0c" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126" "NUMANode"=0
2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:60" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126" "NUMANode"=0
2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:60" "localCPUs"="0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62,64,66,68,70,72,74,76,78,80,82,84,86,88,90,92,94,96,98,100,102,104,106,108,110,112,114,116,118,120,122,124,126" "NUMANode"=0
2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:b4" "localCPUs"="1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127" "NUMANode"=1
2026/03/23 16:03:49 "level"=0 "msg"="PCIE domain" "root"="pci0000:c9" "localCPUs"="1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63,65,67,69,71,73,75,77,79,81,83,85,87,89,91,93,95,97,99,101,103,105,107,109,111,113,115,117,119,121,123,125,127" "NUMANode"=1

Since a CPU might have adjacency to more than one PCIe roots, we can't currently avoid representing the same core in more than one PCIe as CPU isn't in the PCIe device tree. But as you have mentioned, I also believe that kubernetes/enhancements#5942 will help with that as it introduces list-of-*, that should allow us to do better grouping.

[ffromani]

on hold till we rebase on top of kube 1.36 to consume the list attributes