TransPi on HPC failing to pull singularity image

[claumer]

Hi there,

I've been trying to use TransPi (love the concept and the benchmarking in the paper looks great!) to re-assemble some older public datasets. Running on an interactive slurm session of an HPC which has singularity installed:

---

(TransPi) chril1@hpc-gpu-002:~/TransPiDBs$ ./nextflow run ../TransPi/TransPi.nf --all --maxReadLen 150 --k 25,35,55,75,85 --reads '/home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz' --outdir Results_Flagellophora -profile singularity,TransPiContainer

N E X T F L O W ~ version 21.04.1
Launching ../TransPi/TransPi.nf [furious_einstein] - revision: 5a26a6e85d

TransPi - Transcriptome Analysis Pipeline v1.3.0-rc

TransPi.nf Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPi/TransPi.nf
Launch Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs
Results Directory: Results_Flagellophora
Work Directory: /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/work
TransPi DBs: /home/chril1/TransPiDBs
Uniprot DB: /home/chril1/TransPiDBs/DBs/uniprot_db/uniprot_metazoa_33208.fasta
Busco DB: /home/chril1/TransPiDBs/DBs/busco_db/metazoa_odb10
Reads Directory: /home/chril1/NERC_PhyloCourse/transcriptome_data/Flagellophora_sp/SRR8641368_R[1,2].fastq.gz
Read Length: 150
Kmers: 25,35,55,75,85

Running TransPi with your dataset


Running the full TransPi analysis

executor > local (2)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [ 0%] 0 of 1
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (3)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (4)
executor > local (5)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
executor > local (6)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (8)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (9)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (9)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (10)
executor > local (10)
executor > local (10)
executor > local (10)
executor > local (11)
executor > local (11)
executor > local (11)
executor > local (11)
executor > local (11)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
[38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔
[09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔
[a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔
[94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔
[de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1
[- ] process > rna_quast -
[- ] process > mapping_evigene -
[- ] process > busco4 -
[ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1
[- ] process > summary_evigene_individual -
[- ] process > busco4_tri -
[47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔
executor > local (13)
[6c/876909] process > fasqc (SRR8641368_R) [100%] 1 of 1 ✔
[38/2daf75] process > fastp (SRR8641368_R) [100%] 1 of 1 ✔
[09/b36296] process > fastp_stats (SRR8641368_R) [100%] 1 of 1 ✔
[a5/77f26f] process > skip_rrna_removal (SRR8641368_R) [100%] 1 of 1 ✔
[94/71880f] process > normalize_reads (SRR8641368_R) [100%] 1 of 1 ✔
[de/71f78c] process > trinity_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[de/bc6216] process > soap_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[25/457622] process > velvet_oases_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[84/17fe6c] process > rna_spades_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[2e/672733] process > transabyss_assembly (SRR8641368_R) [100%] 1 of 1 ✔
[80/f0f8e6] process > evigene (SRR8641368_R) [ 0%] 0 of 1
[- ] process > rna_quast -
[- ] process > mapping_evigene -
[- ] process > busco4 -
[ca/050e8b] process > mapping_trinity (SRR8641368_R) [ 0%] 0 of 1
[- ] process > summary_evigene_individual -
[- ] process > busco4_tri -
[47/a485e7] process > skip_busco_dist (SRR8641368_R) [100%] 1 of 1, cached: 1 ✔
[- ] process > summary_busco4_individual -
[- ] process > get_busco4_comparison -
[- ] process > transdecoder_longorf -
[- ] process > transdecoder_diamond -
[- ] process > transdecoder_hmmer -
[- ] process > transdecoder_predict -
[- ] process > swiss_diamond_trinotate -
[- ] process > custom_diamond_trinotate -
[- ] process > hmmer_trinotate -
[- ] process > skip_signalP -
[- ] process > skip_tmhmm -
[- ] process > skip_rnammer -
[- ] process > trinotate -
[- ] process > get_GO_comparison -
[- ] process > summary_custom_uniprot -
[- ] process > skip_kegg -
[- ] process > get_transcript_dist -
[- ] process > summary_transdecoder_individual -
[- ] process > summary_trinotate_individual -
[- ] process > get_report -
[98/934751] process > get_run_info [100%] 1 of 1 ✔
Pulling Singularity image docker://null [cache /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/singularityCache/null.img]
Error executing process > 'busco4_tri (SRR8641368_R)'

Caused by:
Failed to pull singularity image
command: singularity pull --name null.img.pulling.1664887974429 docker://null > /dev/null
status : 255
message:
FATAL: While making image from oci registry: error fetching image to cache: failed to get checksum for docker://null: Error reading manifest latest in docker.io/library/null: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

---

It seems that it gets all the way through building the four assemblies then crashes when trying to pull a singularity image for BUSCO evaluation. I confess I'm not quite sure how to debug this one - do you have any ideas? I've also tried -profile conda and am getting environment creation conflict errors out of that, and so seem to have reached an impasse.

Thanks very much for your attention,

Chris L

[sevragorgia]

Hi Chris,

not an expert with the singularity stuff here but it seems as if the image is not being pulled because of some naming problem.

We run TransPi with conda and tend not to get conflicts except for one step. Do you face conflicts creating all environments or only one?

Sergio

[claumer]

Hi Sergio,

Thanks for the fast response. When running the same command in a fresh environment with -profile conda, it halts with the following:

---

Error executing process > 'evigene (SRR8641368_R)'

Caused by:
Failed to create Conda environment
command: conda create --mkdir --yes --quiet --prefix /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/condaEnv/env-0761ea60e49e247680763dddcc8ef240 -c conda-forge bioconda::cd-hit=4.8.1 bioconda::exonerate=2.4 bioconda::blast=2.11.0
status : 1
message:
Building graph of deps: 0%| | 0/8 [00:00<?, ?it/s]
Examining @/linux-64::__cuda==11.6=0: 0%| | 0/8 [00:00<?, ?it/s]
Examining bioconda::blast=2.11.0: 12%|█▎ | 1/8 [00:00<00:00, 8439.24it/s]
Examining @/linux-64::__glibc==2.31=0: 25%|██▌ | 2/8 [00:00<00:00, 131.25it/s]
Examining bioconda::cd-hit=4.8.1: 38%|███▊ | 3/8 [00:00<00:00, 195.25it/s]
Examining @/linux-64::__linux==5.4.0=0: 50%|█████ | 4/8 [00:00<00:00, 110.94it/s]
Examining @/linux-64::__archspec==1=x86_64: 62%|██████▎ | 5/8 [00:00<00:00, 138.28it/s]
Examining bioconda::exonerate=2.4: 75%|███████▌ | 6/8 [00:00<00:00, 165.61it/s]
Examining @/linux-64::__unix==0=0: 88%|████████▊ | 7/8 [00:00<00:00, 79.24it/s]

Determining conflicts:   0%|          | 0/8 [00:00<?, ?it/s]
Examining conflict for exonerate blast __glibc cd-hit:   0%|          | 0/8 [00:00<?, ?it/s]
Examining conflict for blast exonerate cd-hit:  12%|█▎        | 1/8 [00:00<00:00,  7.35it/s]
Examining conflict for blast exonerate cd-hit:  25%|██▌       | 2/8 [00:00<00:00, 14.70it/s]
Examining conflict for blast exonerate:  25%|██▌       | 2/8 [00:00<00:00, 14.70it/s]       
                                                                                     
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versions

Package libzlib conflicts for:
bioconda::cd-hit=4.8.1 -> libzlib[version='>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0']
bioconda::exonerate=2.4 -> glib -> libzlib[version='>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0']
bioconda::cd-hit=4.8.1 -> zlib[version='>=1.2.12,<1.3.0a0'] -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12',build='h36c2ea0_1012|h166bdaf_1014|h166bdaf_0|h166bdaf_4|h166bdaf_3|h166bdaf_2|h166bdaf_1|h36c2ea0_1013']
bioconda::blast=2.11.0 -> curl[version='>=7.77.0,<8.0a0'] -> libzlib[version='1.2.11|1.2.11|1.2.11|1.2.12|1.2.12|1.2.12|1.2.12|1.2.12|>=1.2.11,<1.3.0a0|>=1.2.12,<1.3.0a0',build='h36c2ea0_1012|h166bdaf_1014|h166bdaf_0|h166bdaf_4|h166bdaf_3|h166bdaf_2|h166bdaf_1|h36c2ea0_1013']

Package pcre conflicts for:
bioconda::exonerate=2.4 -> pcre[version='>=8.41,<9.0a0|>=8.43,<9.0a0|>=8.44,<9.0a0|>=8.45,<9.0a0']
bioconda::blast=2.11.0 -> pcre[version='>=8.44,<9.0a0']
bioconda::exonerate=2.4 -> glib -> pcre[version='>=8.42,<9.0a0']

Package libgcc-ng conflicts for:
bioconda::blast=2.11.0 -> bzip2[version='>=1.0.8,<2.0a0'] -> libgcc-ng[version='>=10.3.0|>=12|>=7.3.0|>=9.4.0|>=11.2.0|>=4.9|>=7.2.0']
bioconda::blast=2.11.0 -> libgcc-ng[version='>=7.5.0|>=9.3.0']

Package libstdcxx-ng conflicts for:
bioconda::blast=2.11.0 -> pcre[version='>=8.44,<9.0a0'] -> libstdcxx-ng[version='>=7.3.0']
bioconda::blast=2.11.0 -> libstdcxx-ng[version='>=7.5.0|>=9.3.0']The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__glibc==2.31=0
  - bioconda::blast=2.11.0 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - bioconda::cd-hit=4.8.1 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']
  - bioconda::exonerate=2.4 -> libgcc-ng[version='>=9.3.0'] -> __glibc[version='>=2.17']

Your installed version is: 2.31

Note that strict channel priority may have removed packages required for satisfiability.

[sevragorgia]

Hi Chris,

I checked the code. Specifically, the following line in the error message means that a different docker image is being used in this particular step. Why exactly busco is not included in the transpi image is something I cannot answer because I am not using docker/singularity and cannot test in our hpc anything related to these systems.

Pulling Singularity image docker://null [cache /gpfs/nhmfsa/bulk/share/data/mbl/share/workspaces/users/chril1/TransPiDBs/singularityCache/null.img]

Now, I think you can solve the issue by adding the following line to your nextflow.config file:

TransPiContainer {
        process {
            params.oneContainer = true
            params.TPcontainer="rerv/transpi:v1.0.0"
            //sevra: fix reported problem with busco container
            params.v4container="ezlabgva/busco:v4.0.5_cv1"
        }
    }

the variable params.v4container is used in TransPi in the following block:

if (params.oneContainer){ container "${params.v4container}" } else {
                container (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container ? "https://depot.galaxyproject.org/singularity/busco:4.1.4--py_2" : "ezlabgva/busco:v4.0.5_cv1")
                }

The problem right now is that params.v4container is not defined and set to null. This explains the docker://null call in your error. I guess setting the variable to a valid container will solve the issue; no promises, though.

A working block of code, say Transabyss, looks like:

if (params.oneContainer){ container "${params.TPcontainer}" } else {
            container (workflow.containerEngine == 'singularity' && !params.singularity_pull_docker_container ? "https://depot.galaxyproject.org/singularity/transabyss:2.0.1--pyh864c0ab_7" : "quay.io/biocontainers/transabyss:2.0.1--pyh864c0ab_7")
            }

You see the structure is the same but the TPcontainer is called. This works before params.TPcontainer is set in the config.

I hope this solves your issue. Please report back to the community and feel free to close the issue if solved. Note however, that other issues may arise from using that particular image, I hope not because it is used if the oneContainer variable is false i.e., you are not using one container with all the stuff but getting multiple images for each process in the pipeline.

cheers

Sergio