question about images

[simevo]

This looks interesting, but it's not clear to me in which environment the jobs are executed.
With gitlab runner docker or kubernetes executor, that's the OCI image specified in the image: property of the job; with gitlab runner shell executor that's the host.
In HPC it's very common to use singularity/apptainer images instead of OCI images; is it so that this executor would use the image: property of the job to set the singularity image to run the job into? Thanks

[byjtew]

Good question, we have never tried running a job using a singularity (or other) image with this executor. The given script will be executed on the SSH host just like a user would do it manually. The reason being that we can eliminate any potential side effect on performance due to the containerisation.

Nothing forbids you from running your image manually from the script, but I would not expect the 'image' tag to have any effect (or a failure).

PRs are welcomed though, the code is very simple and it should be fairly feasible to handle the case with a specified 'image'.

[simevo]

Thanks for the quick feedback. I have just created a PR (#7) to add two simple sbatch CLI options that are used in our environment.
ATM I have not yet had a successful job but that's due to the peculiarities of our systems 🤯

In the meantime I am trying to wrap my head around the generated sbatch config file, with this .gitlab-ci.yaml:

---

default:
  tags:
    - slurm

variables:
  CI_SLURM_PARTITION: serial
  CI_SLURM_NNODES: 1
  CI_SLURM_NTASKS: 1
  CI_SLURM_CPUS_PER_TASK: 1
  CI_SLURM_MEM_PER_NODE: 4G
  CI_SLURM_TIMELIMIT: "00-00:10:00"
  CI_SLURM_ACCOUNT: my_accounr
  CI_SLURM_QOS: my_qos

my_serial_slurm_job:
  script:
    - echo "Hello from the Slurm cluster, in job ${SLURM_JOB_NAME}!"
    - touch output_from_parallel_job.txt
  artifacts:
    paths:
      - output_from_parallel_job.txt

we get this:

#!/bin/bash

#SBATCH --job-name=42__114_623
#SBATCH --nodes=1
#SBATCH --mem=4G
#SBATCH --cpus-per-task=1
#SBATCH --ntasks=1
#SBATCH --time=00-00:10:00
#SBATCH --partition=serial
#SBATCH --comment="Automatic job created from GitLab CI"
#SBATCH --chdir=/home/pippo/slurm-gitlab-executor/wd/builds/42__114_623/
#SBATCH --export=ALL
#SBATCH --output=stdout.log
#SBATCH --error=stderr.log
#SBATCH --account=my_account
#SBATCH --qos=my_qos

. /home/pippo/slurm-gitlab-executor/driver/idle_script.sh

I don't quite get the last line! I would have expected it would just dump there the contents of the script key un-yaml'd:

echo "Hello from the Slurm cluster, in job ${SLURM_JOB_NAME}!"
touch output_from_parallel_job.txt

If my simple approach is feasible, to go back to my original question, I'd add a job like:

my_singularity_slurm_job:
  script:
    - srun singularity run docker://hello-world

and that's be it! and BTW to answer my own question, as you say, the image tag would have no effect.

So now my question really is: what do you want to achieve with idle_script.sh? why do you need a "SLURM waiting loop" at all?

[byjtew]

I don't quite get the last line! I would have expected it would just dump there the contents of the script key un-yaml'd:

Just a design choice😄 If you find a good reason to do it a different way I am completely open to it.

If my simple approach is feasible, to go back to my original question, I'd add a job like: ...

That would be the simplest way to do it right now I think, once again, I have never tried doing that so be ready for a bit of debugging. Also feel free to open a PR whenever you'd like if you manage to integrate it.

why do you need a "SLURM waiting loop" at all?

This idle_script.sh is basically a hack to handle edge cases.
It's been a while since I've touched this code but I remember needing when the user cancels the script right after the script started being scheduled, which resulted in a deadlock. I also remember some latency between the time the gitlab executor pasted the files onto the slurm job session, which triggered an immediate halt of the slurm job.

[simevo]

Thanks for the hints, if there is a hack there must a problem it tries to solve! I'll now pause the tests and think about you hack and about the use cases...