Regarding the issue of nvml.init() hanging。

[sakuradiss]

During the use of the device-plugin component, nvml.init() is used to initialize NVML. However, after running stably for more than ten hours, the init function hangs internally. What could be the issue?

Basic Environment:

CentOS 8.8
Go Version: 1.21
go-nvml Version: v0.12.9-0
Driver Version: 570.148.08
CUDA Version: 12.8
GPU: H200

[elezar]

@sakuradiss there is an issue in later versions of the 570.x driver that is being looked internally. Please downgrade to driver version 570.86.15 for now if possible.

[sakuradiss]

@sakuradiss there is an issue in later versions of the 570.x driver that is being looked internally. Please downgrade to driver version 570.86.15 for now if possible.

Thank you for your response. My current solution is to globally initialize nvml.init() only once, which temporarily avoids this issue. However, I haven't used nvml.shutdown(). Could you please explain what risks this approach might entail?

[elezar]

If you don't call Shutdown there is a chance that the resources that are associated with nvml are not cleaned up. I do not have enough insight into the NVML implementation to know whether this would leak resources across application restarts, or whether the resources that are allocated in an Init call would be implicitly cleaned up when the application terminates.