[Bug] MIG destroys are not working correctly at times

[bergentruckung]

I have a 4-way H100 SXM setup and in one of our tools, I'm seeing a weird issue when we destroy MIGs. Destroying CI works fine (default compute instance that uses up all the gpu instance) and then destroying its GI via the API says that they worked, but I see remnants of itself (notice the *):

:sudo nvidia-smi mig -lcip                                                                                                                                                                                                                                        
+--------------------------------------------------------------------------------------+
| Compute instance profiles:                                                           |
| GPU     GPU       Name             Profile  Instances   Exclusive       Shared       |
|       Instance                       ID     Free/Total     SM       DEC   ENC   OFA  |
|         ID                                                          CE    JPEG       |
|======================================================================================|
|   2      3       MIG 1c.2g.20gb       0      2/2           16        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      3       MIG 1c.2g.20gb       7      1/1           26        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      3       MIG 2g.20gb          1*     1/1           32        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      4       MIG 1c.2g.20gb       0      0/2           16        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      4       MIG 1c.2g.20gb       7      0/1           26        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      4       MIG 2g.20gb          1*     0/1           32        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   2      2       MIG 1c.3g.40gb       0      0/3           16        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   2      2       MIG 1c.3g.40gb       7      0/2           26        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   2      2       MIG 2c.3g.40gb       1      0/1           32        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   2      2       MIG 3g.40gb          2*     0/1           60        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   3      3       MIG 1c.2g.20gb       0      0/2           16        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      3       MIG 1c.2g.20gb       7      0/1           26        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      3       MIG 2g.20gb          1*     0/1           32        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      4       MIG 1c.2g.20gb       0      0/2           16        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      4       MIG 1c.2g.20gb       7      0/1           26        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      4       MIG 2g.20gb          1*     0/1           32        2     0     0   |
|                                                                      2     2         |
+--------------------------------------------------------------------------------------+
|   3      2       MIG 1c.3g.40gb       0      0/3           16        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   3      2       MIG 1c.3g.40gb       7      0/2           26        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   3      2       MIG 2c.3g.40gb       1      0/1           32        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+
|   3      2       MIG 3g.40gb          2*     0/1           60        3     0     0   |
|                                                                      3     3         |
+--------------------------------------------------------------------------------------+

:sudo nvidia-smi mig -lgi                                            
+-------------------------------------------------------+            
| GPU instances:                                        |            
| GPU   Name             Profile  Instance   Placement  |            
|                          ID       ID       Start:Size |            
|=======================================================|            
|   2  MIG 2g.20gb         14        3          0:2     |            
+-------------------------------------------------------+            
|   2  MIG 2g.20gb         14        4          2:2     |            
+-------------------------------------------------------+            
|   2  MIG 3g.40gb          9        2          4:4     |            
+-------------------------------------------------------+            
|   3  MIG 2g.20gb         14        3          0:2     |            
+-------------------------------------------------------+            
|   3  MIG 2g.20gb         14        4          2:2     |            
+-------------------------------------------------------+            
|   3  MIG 3g.40gb          9        2          4:4     |            
+-------------------------------------------------------+            
:sudo nvidia-smi mig -lci                                              
+--------------------------------------------------------------------+ 
| Compute instances:                                                 | 
| GPU     GPU       Name             Profile   Instance   Placement  | 
|       Instance                       ID        ID       Start:Size | 
|         ID                                                         | 
|====================================================================| 
|   2      4       MIG 2g.20gb          1         0          0:2     | 
+--------------------------------------------------------------------+ 
|   2      2       MIG 3g.40gb          2         0          0:4     | 
+--------------------------------------------------------------------+ 
|   3      3       MIG 2g.20gb          1         0          0:2     | 
+--------------------------------------------------------------------+ 
|   3      4       MIG 2g.20gb          1         0          0:2     | 
+--------------------------------------------------------------------+ 
|   3      2       MIG 3g.40gb          2         0          0:4     | 
+--------------------------------------------------------------------+ 

Notice how GPU instance 3 is still lying around, even though the API returned successfully. Trying to destroy it by hand also doesn't work:

:sudo nvidia-smi mig -dgi -gi 3 -i 2                                      
Unable to destroy GPU instance ID  3 from GPU  2: In use by another client
Failed to destroy GPU instances: In use by another client                 

I also don't see anything using the corresponding capability device for this GPU instance (I don't want to do the same on /dev/nvidia2 as that has other MIGs on which processes would be running):

:grep gpu2/gi3 /proc/driver/nvidia-caps/mig-minors  
gpu2/gi3/access 300                                 
gpu2/gi3/ci0/access 301                             
gpu2/gi3/ci1/access 302                             
gpu2/gi3/ci2/access 303                             
gpu2/gi3/ci3/access 304                             
gpu2/gi3/ci4/access 305                             
gpu2/gi3/ci5/access 306                             
gpu2/gi3/ci6/access 307                             
gpu2/gi3/ci7/access 308                             
:lsof /dev/nvidia-caps/nvidia-cap300                

What is happening here?

I'm using Nvidia driver 550.144.03, CUDA rt 12.2.140, go-nvml v0.12.4-0 and on Go 1.22.5.

Thanks in advance.

[bergentruckung]

I still see this happen. I'm able to destroy the compute instance, but destroying the GPU instance fails.

I have a live instance if you need additional logs. Let me know.

[klueska]

The * on the ComputeInstance indicates that the ComputeInstance actually wasn't actually destroyed (which is then why the GpuInstance isn't destroyable). I don't have a quick answer as to why the ComputeInstance wasn't destroyed properly, but that is where your debugging should be focused, not on the GpuInstance.

Actually, ignore what I said above.

So, just to be clear, you are referring to this GI, i.e. GI 3 from Profile 14 on GPU 2:

|   2  MIG 2g.20gb         14        3          0:2     |  

[klueska]

What environment are you running in? Is this a bare-metal setup or with Kubernetes? Is there anything at all that might still have a handle open on the GI, such as the kubernetes device plugin, the nvidia-mig-manager, the dcgm-exporter, or any other tools such as these?

Just checking :lsof /dev/nvidia-caps/nvidia-cap300 isn't sufficient as this device node is just used for access control (i.e. it's not held open even if something else still has a handle on the GI).

[bergentruckung]

So, just to be clear, you are referring to this GI, i.e. GI 3 from Profile 14 on GPU 2:

Yup, correct.

What environment are you running in? Is this a bare-metal setup or with Kubernetes? Is there anything at all that might still have a handle open on the GI, such as the kubernetes device plugin, the nvidia-mig-manager, the dcgm-exporter, or any other tools such as these?

It's a bare-metal setup. We use DCGM's Go bindings to dynamically add/remove devices. But, that seems to work as expected. Is there any way to figure out what process has open file handles to a GI (similar in spirits to lsof /dev/nvidia*).

Thanks!

[bergentruckung]

Also, can the handle to a CI change during its lifetime? Based on NVML API docs, it shouldn't:

Compute instance IDs are unique per GPU instance and remain valid until the compute instance is destroyed.

But, I'm seeing a case when I try to destroy a CI, it gives me nvml.ERROR_INVALID_ARGUMENT. When can this happen? Just noting over here that there's no other code block in my app that tries to destroy this CI, to rule out other threads/processes.

[bergentruckung]

@klueska Any pointers on what I can check from my side? Based on what I see, there shouldn't be any pids with open file handles to the CI. What else could be preventing us from destroying this CI?