Skip to content

Fix containers losing access to NVIDIA GPUs #134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hippogr wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

Fix containers losing access to NVIDIA GPUs #134

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/rest-server/src/models/v2/job/k8s.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -550,7 +550,7 @@ const generateTaskRole = (
},
],
securityContext: {
...((dindMode) && { privileged: true }),
privileged: true,
Copy link

Copilot AI Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting privileged to true unconditionally for all user jobs poses a significant security risk. Privileged containers have access to all host devices and can bypass most container security features, essentially granting root-level access to the host system. The previous implementation only enabled privileged mode when dindMode (Docker-in-Docker) was active, which was a more secure approach.

While privileged mode may enable GPU access, it's important to verify that this is truly necessary. NVIDIA GPUs can typically be accessed in containers without privileged mode when the NVIDIA Container Runtime is properly configured. Consider investigating whether the issue is actually with the container runtime configuration, device plugin setup, or other aspects of the GPU passthrough mechanism rather than requiring privileged containers.

Suggested change
privileged: true,
...(launcherConfig.dindMode && { privileged: true }),

Copilot uses AI. Check for mistakes.
capabilities: {
add: ['SYS_ADMIN', 'IPC_LOCK', 'DAC_READ_SEARCH'],
drop: ['MKNOD'],
Expand Down