SOS! Help! I cannot get enroot/pyxis to work: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set

[tjoymeed]

Hi there,

SOS! Could you please kindly help me? I just couldn't get pyxis work:

Thanks!

---

enroot import 'docker://nvcr.io#nvidia/nemo:25.07.gpt_oss'

enroot create --name pyxis_nemo-25.07.gpt_oss nvidia+nemo+25.07.gpt_oss.sqsh

And then since then I updated one of the sub-folders inside the container slightly by using the latest repo main branch, so I am not willing to directly to pull from NVIDIA registry in srun.

mike@ln01:~/enroot$ enroot list
cache
data
nemo-25.04
pyxis_hiyouga
pyxis_nemo-25.07.gpt_oss
pyxis_nemo-25.07.gpt_oss.sqsh
mike@ln01:~/enroot$ enroot version
3.5.0
mike@ln01:~/enroot$ ll
total 15122412
drwxr-xr-x 2 mike domain users        4096 Sep 25 17:05 ./
drwx------ 2 mike domain users        4096 Sep 25 16:43 ../
drwx------ 2 mike domain users        4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users        4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users        4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users        4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users        4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh


**mike@ln01:~/enroot$ srun --partition=gpu-interactive --account=gpu   --container-name="nemo-25.07.gpt_oss"   /bin/hostname**
srun: job 1276806 queued and waiting for resources
srun: job 1276806 has been allocated resources
slurmstepd-gn12: error: pyxis: error: a container with name "pyxis_nemo-25.07.gpt_oss" does not exist, and --container-image is not set
slurmstepd-gn12: error: spank: required plugin spank_pyxis.so: task_init() failed with rc=-1
slurmstepd-gn12: error: Failed to invoke spank plugin stack
srun: error: gn12: task 0: Exited with exit code 1



mike@ln01:~/enroot$ ll
total 15122412
drwxr-xr-x 2 mike domain users        4096 Sep 25 17:26 ./
drwx------ 2 mike domain users        4096 Sep 25 16:43 ../
drwx------ 2 mike domain users        4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users        4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users        4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users        4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users        4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh
mike@ln01:~/enroot$ echo $ENROOT_DATA_PATH 
/home/mike/enroot

[krono]

This is not how the enroot/pyxis interaction will work.
If you want an enroot container that is also avaiable via pyxis, you have to lookup how pyxis is configured in your cluster.

[tjoymeed]

What do you mean?

Note the following had no problem:

mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=docker://ubuntu:22.04 /bin/echo "Pyxis works"

srun: job 1276824 queued and waiting for resources

srun: job 1276824 has been allocated resources

pyxis: importing docker image: docker://ubuntu:22.04

pyxis: imported docker image: docker://ubuntu:22.04

Pyxis works

[krono]

It does, but pyxis configures a lot of $ENROOT vars differenly, you should ask there probably…

[tjoymeed]

I have to get the local updated unpacked rootfs folder working (since I did the enroot import and then enroot create and then did some local updates already...)

I have also made the unpacked rootfs folder into a squashfs file:

Create a squashfs file from your unpacked rootfs

mksquashfs pyxis_nemo-25.07.gpt_oss pyxis_nemo-25.07.gpt_oss.sqsh -noappend

when I use this image file, it hung:

mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=/home/mike/enroot/pyxis_nemo-25.07.gpt_oss.sqsh /bin/echo "Pyxis works"
srun: job 1276833 queued and waiting for resources
srun: job 1276833 has been allocated resources

[* HUNG! I had to type CRTL+C to stop it *]

^Csrun: interrupt (one more within 1 sec to abort)
srun: StepId=1276833.0 task 0: running
^Csrun: sending Ctrl-C to StepId=1276833.0
srun: forcing job termination
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
slurmstepd-gn12: error: *** STEP 1276833.0 ON gx12 CANCELLED AT 2025-09-25T18:09:16 ***
slurmstepd-gn12: error: pyxis: child 1070021 terminated with signal 9
^Csrun: sending Ctrl-C to StepId=1276833.0
srun: job abort in progress

---

I recall that when I created it, it had some warning message:
Not sure if this warning message is the problem:

Parallel mksquashfs: Using 96 processors
Creating 4.0 filesystem on pyxis_nemo-25.07.gpt_oss.sqsh, block size 131072.
Could not open pyxis_nemo-25.07.gpt_oss/etc/slurm, skipping...

---

[tjoymeed]

It does, but pyxis configures a lot of $ENROOT vars differenly, you should ask there probably…

Thanks! Okay I added: export ENROOT_RUNTIME_PATH="$HOME/.local/run/enroot"

But still got the same issues...

[krono]

That is expected. pyxis actively ignores most $ENROOT_ variables.

Maybe this will work?

enroot import ....
enroot create ...
enroot export ... image.sqsh
mv image.sqsh $GLOBAL_STORAGE
srun ... --container-image=$GLOBAL_STORAGE/image.sqsh ...

point is the export.

[tjoymeed]

Thanks a lot!

Yes, the enroot export is better than the mksquashfs I've used yesterday, which had a warning:

Today enroot export is cleaner, with no warning.

Still it hung, what's the problem?

---

mike@ln01:~/enroot$ ll
total 15122412
drwxr-xr-x 2 mike domain users 4096 Sep 25 17:26 ./
drwx------ 2 mike domain users 4096 Sep 25 18:24 ../
drwx------ 2 mike domain users 4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users 4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users 4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users 4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users 4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh

mike@ln01:~/enroot$ enroot export --output test.sqsh pyxis_nemo-25.07.gpt_oss
[INFO] Creating squashfs filesystem...

Parallel mksquashfs: Using 96 processors
Creating 4.0 filesystem on /sc/home/mike/enroot/test.sqsh, block size 131072.
[===============================================================================================================================================================================================================] 529472/529472 100%

Exportable Squashfs 4.0 filesystem, lzo compressed, data block size 131072
uncompressed data, compressed metadata, compressed fragments,
compressed xattrs, compressed ids
duplicates are removed
Filesystem size 29009641.95 Kbytes (28329.73 Mbytes)
89.59% of uncompressed filesystem size (32380218.35 Kbytes)
Inode table size 4553800 bytes (4447.07 Kbytes)
30.98% of uncompressed inode table size (14698293 bytes)
Directory table size 4437568 bytes (4333.56 Kbytes)
38.26% of uncompressed directory table size (11597683 bytes)
Number of duplicate files found 75371
Number of inodes 377279
Number of files 310435
Number of fragments 20631
Number of symbolic links 21513
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 45331
Number of hard-links 6
Number of ids (unique uids + gids) 1
Number of uids 1
root (0)
Number of gids 1
root (0)
mike@ln01:/enroot$
mike@ln01:/enroot$
mike@ln01:~/enroot$ ll
total 44132056
drwxr-xr-x 2 mike domain users 4096 Sep 25 18:40 ./
drwx------ 2 mike domain users 4096 Sep 25 18:24 ../
drwx------ 2 mike domain users 4096 Jun 30 11:48 cache/
drwx------ 2 mike domain users 4096 Sep 25 16:21 data/
drwxr-xr-x 2 mike domain users 4096 Jul 14 01:33 nemo-25.04/
drwxr-xr-x 2 mike domain users 4096 May 11 03:21 pyxis_hiyouga/
drwxrwxrwx 2 mike domain users 4096 Sep 21 17:57 pyxis_nemo-25.07.gpt_oss/
-rw-r----- 1 mike domain users 15485349888 Sep 24 22:53 pyxis_nemo-25.07.gpt_oss.sqsh
-rw-r----- 1 mike domain users 29705875456 Sep 25 19:14 test.sqsh

mike@ln01:~/enroot$ srun --nodes=1 --ntasks=1 --partition=gpu-interactive --account=gpu --container-image=/home/mike/enroot/test.sqsh /bin/echo "Pyxis works"

srun: job 1277196 queued and waiting for resources
srun: job 1277196 has been allocated resources

[HUNG HERE! I HAD TO CTRL+C]

^Csrun: interrupt (one more within 1 sec to abort)
srun: StepId=1277196.0 task 0: running
^Csrun: sending Ctrl-C to StepId=1277196.0
srun: forcing job termination
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
slurmstepd-gn11: error: *** STEP 1277196.0 ON gn11 CANCELLED AT 2025-09-25T19:18:52 ***
slurmstepd-gn11: error: pyxis: child 1231834 terminated with signal 9

[tjoymeed]

That is expected. pyxis actively ignores most $ENROOT_ variables.

Maybe this will work?

enroot import ....
enroot create ...
enroot export ... image.sqsh
mv image.sqsh $GLOBAL_STORAGE
srun ... --container-image=$GLOBAL_STORAGE/image.sqsh ...

point is the export.

Thanks a lot!

My home directory is shared anyway on the compute node. Is it the $GLOBAL_STORAGE you meant?

[krono]

Yes.
Anyway, further problems should probably reportet at https://github.com/NVIDIA/pyxis