fix: handle GetMemoryInfo NOT_SUPPORTED for unified memory GPUs (e.g., NVIDIA GB10)

[jsl9208]

Summary

Fixes #1511

The NVIDIA GB10 (DGX Spark) uses a unified memory architecture where nvmlDeviceGetMemoryInfo() returns ERROR_NOT_SUPPORTED. This causes register.go to panic(0), crashing the device plugin on these GPUs.

Changes

• register.go: Handle ERROR_NOT_SUPPORTED gracefully with a switch/case. Fall back to a new DefaultDeviceMemory config value (in MB). If not configured, skip the device with an error log instead of panicking.
• metrics.go: Skip memory metrics collection when ERROR_NOT_SUPPORTED is returned, instead of treating it as a fatal error.
• device.go: Add DefaultDeviceMemory int64 field to NvidiaConfig.
• Helm chart: Plumb defaultDeviceMemory through configmap and values.yaml (default: 0).

Usage

For unified memory GPUs, users must set defaultDeviceMemory to the total GPU memory in MB:

# values.yaml
devicePlugin:
  defaultDeviceMemory: 131072  # 128 GB for GB10

Without this config, the device will be skipped (not registered) but the plugin won't crash.

Testing

Tested on real hardware: NVIDIA DGX Spark (GB10, ARM64, Driver 580.95.05, CUDA 13.0).

Scenario	Result
Unpatched v2.8.0 on GB10	panic: 0 at register.go:115 ❌
Patched v2.8.0 with defaultDeviceMemory: 131072	Device registered, pod scheduled, vGPU memory isolation works ✅
Patched v2.8.0 without defaultDeviceMemory	Device skipped gracefully, no crash ✅

[hami-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jsl9208
Once this PR has been reviewed and has the lgtm label, please assign wawa0210 for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hami-robot]

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.

📝 Please follow instructions in the contributing guide to update your commits with the DCO

Full details of the Developer Certificate of Origin can be found at developercertificate.org.

The list of commits missing DCO signoff:

• c301f9b Add unit tests for cambricon's Fit Function (Add unit tests for cambricon's Fit Function #1198)
• 5dd402f Add Ascend unit tests, mainly for Fit Function (Add unit tests for Ascend's Fit Function #1197)
• 56ffbcb refactor: clean up code and improve maintainability (refactor: clean up code and improve maintainability #1195)
• 0aa7bb8 fix: benchmarks/ai-benchmark/Dockerfile to reduce vulnerabilities ([Snyk] Security upgrade tensorflow/tensorflow from latest-gpu to 2.20.0rc0-gpu #1243)
• 1591156 fix: handle GetMemoryInfo NOT_SUPPORTED for unified memory GPUs

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hami-robot]

Welcome @jsl9208! It looks like this is your first PR to Project-HAMi/HAMi 🎉

[gemini-code-assist]

Summary of Changes

Hello @jsl9208, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces critical stability improvements for the NVIDIA device plugin, specifically addressing issues with GPUs that utilize a unified memory architecture, such as the NVIDIA GB10. The changes ensure that the device plugin can operate reliably on such hardware by providing a configurable fallback for device memory information or by gracefully handling unsupported memory queries, thus preventing application crashes and enhancing overall system robustness.

Highlights

• Unified Memory GPU Support: The device plugin now gracefully handles nvmlDeviceGetMemoryInfo() returning ERROR_NOT_SUPPORTED for unified memory GPUs (e.g., NVIDIA GB10), preventing crashes.
• Configurable Default Memory: A new DefaultDeviceMemory configuration value (in MB) has been introduced. If ERROR_NOT_SUPPORTED occurs, this value is used as a fallback for device memory.
• Graceful Device Skipping: If DefaultDeviceMemory is not configured for unified memory GPUs that return ERROR_NOT_SUPPORTED, the device will be skipped with an error log instead of causing a panic, ensuring plugin stability.
• Metrics Collection Adjustment: Memory metrics collection is now skipped when ERROR_NOT_SUPPORTED is returned by nvmlDeviceGetMemoryInfo(), preventing fatal errors during monitoring.
• Helm Chart Integration: The defaultDeviceMemory parameter has been integrated into the Helm chart's configmap and values.yaml for easy configuration.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files

• Ignored by pattern: .github/workflows/** (16)
  • .github/workflows/auto-label-pr.yaml
  • .github/workflows/auto-release.yaml
  • .github/workflows/call-e2e-upgrade.yaml
  • .github/workflows/call-e2e.yaml
  • .github/workflows/call-release-helm.yaml
  • .github/workflows/call-release-image-hamicore.yaml
  • .github/workflows/call-release-image.yaml
  • .github/workflows/call-release-notes.yaml
  • .github/workflows/call-release-website.yaml
  • .github/workflows/ci-image-scanning.yaml
  • .github/workflows/ci.yaml
  • .github/workflows/codeql-analysis.yml
  • .github/workflows/issue-translate.yaml
  • .github/workflows/lint-chart.yaml
  • .github/workflows/stale.yaml
  • .github/workflows/test-self-hosted.yaml

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a crucial fix for GPUs with unified memory architecture by gracefully handling cases where memory information is not available, preventing panics in the device plugin. The addition of the defaultDeviceMemory configuration provides a flexible fallback mechanism.
The PR also includes a significant number of repository improvements, such as adding comprehensive documentation, issue templates, CI/CD configurations with golangci-lint and dependabot, and a new Helm chart. These changes greatly enhance the project's maintainability and developer experience.
I've added one comment regarding a potential division-by-zero issue in the metrics collection logic for safer metric reporting. Overall, this is an excellent and substantial contribution to the project.

[gemini-code-assist]

To prevent a potential division-by-zero if devs.Device.Totalmem is 0, it's safer to add a check before performing the division. A zero value could lead to NaN or Inf values in your metrics, which might cause issues with monitoring and alerting systems.

var memPercentage float64
if devs.Device.Totalmem > 0 {
	memPercentage = float64(devs.Device.Usedmem) / float64(devs.Device.Totalmem)
}
ch <- prometheus.MustNewConstMetric(
	nodeGPUMemoryPercentage,
	prometheus.GaugeValue,
	memPercentage,
	nodeID, devs.Device.ID, fmt.Sprint(devs.Device.Index),
)