Set proper path for MPS shm dir mount

[kasia-kujawa]

Fixes #974

This unblocks usage of MPS on GPUs which support pre-Volta MPS (e.g. NVIDIA P4) on GKE clusters and it doesn't change default configuration.

The MPS control daemon template hardcoded /driver-root/dev/shm as the container mount path for the MPS shm directory. This works when the daemon runs inside a chroot (standard NVIDIA DRA driver install, GPU Operator),
but fails on GKE COS where the daemon runs directly in the container namespace, expecting /dev/shm.

Change was also tested on NVIDIA T4 which supports Volta MPS.

1. volumeMounts on EKS, MPS control daemon uses chroot

      volumeMounts:
        - name: driver-root
          mountPath: /driver-root
        - name: mps-shm-directory
          mountPath: /driver-root/dev/shm
        - name: mps-pipe-directory
          mountPath: /driver-root/tmp/nvidia-mps
        - name: mps-log-directory
          mountPath: /driver-root/var/log/nvidia-mps
        - name: kube-api-access-b7kx9
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount

2. volumeMounts on GKE, MPS control daemon doesn't use chroot

      volumeMounts:
        - name: driver-root
          mountPath: /driver-root
        - name: mps-shm-directory
          mountPath: /dev/shm
        - name: mps-pipe-directory
          mountPath: /driver-root/tmp/nvidia-mps
        - name: mps-log-directory
          mountPath: /driver-root/var/log/nvidia-mps
        - name: kube-api-access-7pbnm
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          readOnly: true
          recursiveReadOnly: Disabled

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[shivamerla]

Good catch! The dev root changes as per the driver installation directory in other cases(host installed driver or the driver container), but not for GKE.

[visheshtanksale]

Thanks for looking into this.

[kasia-kujawa]

I have another idea for fixing this issue, shm mount path can be set in this way:

func setMpsShmMountPath(hostDriverRoot string) string {
       for _, sh := range []string{"/bin/sh", "/usr/bin/sh"} {
               if _, err := os.Stat(filepath.Join(hostDriverRoot, sh)); err == nil {
                       return filepath.Join(hostDriverRoot, "dev", "shm")
               }
       }
       return "/dev/shm"
}

and

func (m *MpsControlDaemon) Start(ctx context.Context, config *configapi.MpsConfi
                MpsLogDirectory:                 m.logDir,
                MpsImageName:                    m.manager.config.flags.imageName,
                FeatureGates:                    featuregates.ToMap(),
-               MpsShmMountPath:                 m.manager.mpsShmMountPath,
+               MpsShmMountPath:                 setMpsShmMountPath(m.manager.hostDriverRoot),
        }

This will mirror the logic in the template https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/476b4787f1f75bb4a9c0b8e20604125536514b8c/templates/mps-control-daemon.tmpl.yaml#L32

If you think that it is better idea I can test it tomorrow.

[shivamerla]

Yes, validating whether driverRoot is actually a proper root filesystem seems like a better approach than introducing yet another Helm variable for this.

[k8s-triage-robot]

Unknown CLA label state. Rechecking for CLA labels.

Send feedback to sig-contributor-experience at kubernetes/community.

/check-cla
/easycla

[kasia-kujawa]

@shivamerla approach was changed and it is ready for review.
I tested this on EKS and GKE.

[kasia-kujawa]

@shivamerla this is now ready for next review iteration

[shivamerla]

LGTM

[shivamerla]

@dims please review and provide approval as well.

[dims]

thanks for the very focused fix @kasia-kujawa

[dims]

/approve
/lgtm

[dims]

@kasia-kujawa do you mind fixing up the linting issue?

run golangci-lint
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint config path] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint config verify] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint run  -v --timeout 5m] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Error: cmd/gpu-kubelet-plugin/sharing.go:59:2: Comment should end in a period (godot)
  	// driverRootMountDir is the directory where the driver root is mounted inside the kubelet plugin container
  	^
  1 issues:
  * godot: 1

[kasia-kujawa]

@kasia-kujawa do you mind fixing up the linting issue?

run golangci-lint
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint config path] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint config verify] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Running [/home/runner/golangci-lint-2.11.4-linux-amd64/golangci-lint run  -v --timeout 5m] in [/home/runner/work/nvidia-dra-driver-gpu/nvidia-dra-driver-gpu] ...
  Error: cmd/gpu-kubelet-plugin/sharing.go:59:2: Comment should end in a period (godot)
  	// driverRootMountDir is the directory where the driver root is mounted inside the kubelet plugin container
  	^
  1 issues:
  * godot: 1

@dims I added the missing dot, hope that now it will be ok 🤞 😄

[dims]

/approve
/lgtm

🤞🏾 @kasia-kujawa :)

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dims, kasia-kujawa, shivamerla

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dims]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment