SLURM job time configuration in CLI mode

[mgharawy]

I have been trying to launch an example job on a slurm cluster in CLI mode.
Slurm is configured to require --time to be specified in all jobs, so when the pyp tries to start workers the sbatch command fails because --time wasn't specified.

Error message:

sbatch: error: Please specify --time= for your job.
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit).

I tried adding it to the pyp command as shown below but it wasn't successful

cd nextPyP
module load singularity
export PYP_CONFIG=${PWD}/config.toml
./pyp \
    -data_mode spr                                  \
    -scope_voltage 300                              \
    -scope_pixel 0.66                               \
    -data_path="./T20S/nextpyp_spr_tutorial/14*.tif"       \
    -gain_reference="./T20S/nextpyp_spr_tutorial/Gain.mrc" \
    -gain_flipv                                     \
    -detect_rad 80                                  \
    -detect_method all                              \
    -detect_dist 40                                 \
    -slurm_tasks 7                                  \
    -slurm_memory 70                                \
    -slurm_merge_tasks 7                            \
    -slurm_merge_memory 70			    \
    -slurm_merge_walltime 00:20:00                \
    -slurm_verbose                                \
    -slurm_profile

I have also tried using -slurm_walltime instead and got same error.

My config.toml file:

[pyp]

# path to the PYP container file, from a SLURM compute node's point of view
container = '/home/mgh/nextPyP/pyp.sif'

# fast storage for temporary files, ideally local to the compute node
scratch = '/home/mgh/tmp'

# Bind network locations as needed
binds = []
#containerExec = { module = 'singularity', exec = 'singularity' }
containerExec = '/sw/rl9g/singularity/3.9.7/rl9_go1.19.4/install/bin/singularity'

[slurm]
# user account for the SLURM login node
user = 'mgh'

# hostname for the SLURM login node
host = 'login.node'

# configure SLURM partitions here
queues = [ "batch" ]

# path to SLURM executables
path = '/opt/slurm/cluster/install/RedHat-9/bin/'

Is there another way to configure it in CLI?

[abartesaghi]

What was the exact command that included the --slurm_walltime option? nextPYP should always use the --time option when submitting jobs to slurm. If the --slurm_waltime option is not specified, a default value 1-00:00:00 would be used, maybe that value exceeds some limit on your setup?

[mgharawy]

Hi,
The exact command was the following:

The time limit for a job on the cluster is 14 days, so i don't believe it was exceeding it.

./pyp \
    -data_mode spr                                                                   \
    -scope_voltage 300                                                             \
    -scope_pixel 0.66                                                                \
    -data_path="./T20S/nextpyp_spr_tutorial/14*.tif"              \
    -gain_reference="./T20S/nextpyp_spr_tutorial/Gain.mrc" \
    -gain_flipv                                            \
    -detect_rad 80                                      \
    -detect_method all                               \
    -detect_dist 40                                      \
    -slurm_tasks 7                                       \
    -slurm_memory 70                                \
    -slurm_account mgh                             \
    -slurm_walltime 00-02:00:00                 \
    -slurm_tasks 4                                       \
    -slurm_queue batch                              \
    -slurm_verbose                                      \
    -slurm_profile

[abartesaghi]

The command you’re using looks good and should correctly call sbatch with the --time 00-02:00:00 option on your system. However, there’s a second job, pyp_sprmerge, which I’ve confirmed isn’t passing the -slurm_merge_walltime parameter to sbatch via the --time option. That’s likely what’s causing the error you’re seeing. This is a bug.

I’ve just updated the installation files for version 0.7.0. If you reinstall that version, the issue should be resolved. We’ve also added more detailed debug output when using the -slurm_verbose option, so you can now see exactly which options are passed to sbatch for each job.

[mgharawy]

Thanks, i have tried and it can now submit the jobs.
However, they fail after a few seconds with the following errors.

2025-05-29 11:39:54 [INFO] /opt/pyp/bin/run/pyp:3988 | Job 157 running nextPYP v0.7.1 on amd-dy-intel-1 using 6 task(s) and 24 GB of RAM
ERROR: Cannot find configuration file /home/mgh/.pyp/config.toml
2025-05-29 11:39:54 [ERROR] /opt/pyp/bin/run/pyp:166 |
  ERROR: pyp exiting at 2025-05-29 11:39:54
Traceback (most recent call last):
  File "/opt/pyp/bin/run/pyp", line 5875, in <module>
    trackback()
  File "/opt/pyp/bin/run/pyp", line 3584, in trackback
    get_free_space(Path(os.environ["PYP_SCRATCH"]).parents[0])
  File "/opt/conda/envs/pyp/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'PYP_SCRATCH'

I have exported PYP_SCRATCH in .bashrc to point to the local storage on compute node, also tried setting it to a directory under /home/mgh but got the same errors.

Also exported PYP_CONFIG to point to the config file in my home directory.

#export PYP_SCRATCH=/dev/shm
export PYP_SCRATCH=/home/mgh/tmp
export PYP_CONFIG=/home/mgh/pyp/config.toml

[abartesaghi]

The location of the scratch directory is defined by the scratch setting in the configuration file, so there's no need to manually set PYP_SCRATCH. The error suggests that pyp can't locate your configuration file. It first checks the path specified by PYP_CONFIG, and if that fails, it falls back to the default location: /home/mgh/.pyp/config.toml. If PYP_CONFIG is correctly set (i.e., it points to a valid file and is properly defined in your .bashrc), one possible issue is that your home directory isn't mounted into the container, making it inaccessible to pyp. To resolve this, update the following line in your configuration file:

# Bind network locations as needed
binds = []

to:

# Bind network locations as needed
binds = [ '/home/mgh/pyp' ]

Alternatively, you can move your configuration file to the default location at /home/mgh/.pyp/config.toml. This way, no changes to the configuration file itself are needed.