Nextflow reports (resource usage) not working

[conjon42]

Description of the bug

On all runs of this pipeline, I have not been able to get resource usage metrics in the trace or report files. Upon inspecting the command.log files, I find this error when initiating:

command -v ps &>/dev/null || { >&2 echo "Command 'ps' required by nextflow to collect task metrics cannot be found"; exit 1; 

I suspect this is due to some configuration with the containers used, perhaps they come with a very bare linux distro that does not include these monitoring tools. From the Nextflow documentation on reports:

Nextflow collects trace metrics through a background process for each job in the target environment. Make sure the following tools are available in the environment where tasks are executed: awk, date, grep, ps, sed, tail, tee. Container images that do not include all of these tools may lead to missing data.

I use the singularity containers on HPC. Other pipelines have been able to output these metrics with similarly-sourced containers (Biocontainers and Galaxy Project distributions), but I have not tested any yet that use the exact same tools as used in this pipeline. I will see if I can run test suites on some pipelines that use the same nf-core modules to see if the problem is specific to this pipeline or if it is somewhere higher on the chain.

[nschan]

Hi,

Thanks for reporting this.
It is indeed a bit strange that you cannot get the resource reports. As you note correctly, since this happens inside the container, the container needs ps, etc.

Is this a specific module that is causing this, or is it all of them?

[conjon42]

It appears to be all of them. My reports include the start and end time for each step, but not any of the other resource usages. I pulled down a few of the specific containers called by the nf-core modules used in this pipeline, and tested them to see if they could use tools like ps, grep, etc. For example, the command: singularity exec https://depot.galaxyproject.org/singularity/hifiasm:0.25.0--h5ca1c30_0 succeeds when used in isolation on my HPC. I am using nextflow version 25.04.2 with this pipeline.

[nschan]

So you can use e.g. ps via singularity exec in the container, but nextflow cannot? That is kind of strange, especially if this problem is pipeline specific. I don't know what is causing this, but I will look into it.