Manta build failure

[pontushojer]

Seems there is a problem with the manta bioconda build, see discussion in bioconda/bioconda-recipes#53419 (relates to update in bioconda/bioconda-recipes#52517)

configManta.py --bam final.phased.cram --referenceFasta /scratch/user/ajd/Serina/TellSeq2/ref_genomes/S_tuberosum_v6.1/Stuberosum_686_v6.1.fa --callRegions primary.bed.gz --runDir manta_tmp &> manta.log && python2 manta_tmp/runWorkflow.py --jobs 20 &>> manta.log && convertInversion.py $(which samtools) /scratch/user/ajd/Serina/TellSeq2/ref_genomes/S_tuberosum_v6.1/Stuberosum_686_v6.1.fa manta_tmp/results/variants/diploidSV.vcf.gz 2> manta.log | bgzip -c > manta_tmp/results/variants/diploidSV.convertInversion.vcf.gz && cp manta_tmp/results/variants/diploidSV.convertInversion.vcf.gz final.manta_sv_calls.vcf.gz && cp manta_tmp/results/variants/candidateSmallIndels.vcf.gz final.manta_cand_small.vcf.gz && cp manta_tmp/results/variants/candidateSmallIndels.vcf.gz.tbi final.manta_cand_small.vcf.gz.tbi && cp manta_tmp/results/variants/candidateSV.vcf.gz final.manta_cand_sv.vcf.gz && cp manta_tmp/results/variants/candidateSV.vcf.gz.tbi final.manta_cand_sv.vcf.gz.tbi
Activating conda environment: /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75
[Wed Jan 29 07:42:21 2025]
Error in rule manta:
jobid: 551
output: final.manta_sv_calls.vcf.gz, final.manta_cand_small.vcf.gz, final.manta_cand_small.vcf.gz.tbi, final.manta_cand_sv.vcf.gz, final.manta_cand_sv.vcf.gz.tbi, manta_tmp, manta_tmp/runWorkflow.py
log: manta.log (check log file(s) for error message)
conda-env: /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75
shell:
configManta.py --bam final.phased.cram --referenceFasta /scratch/user/ajd/Serina/TellSeq2/ref_genomes/S_tuberosum_v6.1/Stuberosum_686_v6.1.fa --callRegions primary.bed.gz --runDir manta_tmp &> manta.log && python2 manta_tmp/runWorkflow.py --jobs 20 &>> manta.log && convertInversion.py $(which samtools) /scratch/user/ajd/Serina/TellSeq2/ref_genomes/S_tuberosum_v6.1/Stuberosum_686_v6.1.fa manta_tmp/results/variants/diploidSV.vcf.gz 2> manta.log | bgzip -c > manta_tmp/results/variants/diploidSV.convertInversion.vcf.gz && cp manta_tmp/results/variants/diploidSV.convertInversion.vcf.gz final.manta_sv_calls.vcf.gz && cp manta_tmp/results/variants/candidateSmallIndels.vcf.gz final.manta_cand_small.vcf.gz && cp manta_tmp/results/variants/candidateSmallIndels.vcf.gz.tbi final.manta_cand_small.vcf.gz.tbi && cp manta_tmp/results/variants/candidateSV.vcf.gz final.manta_cand_sv.vcf.gz && cp manta_tmp/results/variants/candidateSV.vcf.gz.tbi final.manta_cand_sv.vcf.gz.tbi
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Looking specifically within the manta.log file, there was a repeated line that mentioned the following issue with the getAlignmentStats_generateStats_000 task:

```[2025-01-29T13:39:03.556938Z] [c315.cluster] [154838_1] [TaskRunner:getAlignmentStats_generateStats_000] [WARNING] Task: 'getAlignmentStats_generateStats_000' failed but qualifies for retry. Total task failures (including this one): 1. Task command: '/scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75/libexec/GetAlignmentStats --ref /scratch/user/ajd/Serina/TellSeq2/ref_genomes/S_tuberosum_v6.1/Stuberosum_686_v6.1.fa --output-file /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/manta_tmp/workspace/alignmentStats.xml.tmpdir/alignmentStats.xml.000.xml --align-file /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/final.phased.cram'```

The final line of manta.log contained the following:
`
[2025-01-29T13:42:21.463944Z] [c315.cluster] [154838_1] [WorkflowRunner] [ERROR] [2025-01-29T13:42:05.043215Z] [c315.cluster] [154838_1] [getChromDepth_sample000_estimateChromDepth_245_scaffold_355_to_261_scaffold_373] /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75/libexec/GetChromDepth: error while loading shared libraries: libboost_filesystem.so.1.85.0: cannot open shared object file: No such file or directory`

I'm attaching both log files in case these are helpful at all. I am happy to provide any other details that may be of assistance. Thanks again for your help!

[tellseq_ktz1.txt](https://github.com/user-attachments/files/18589334/tellseq_ktz1.txt)
[manta.log](https://github.com/user-attachments/files/18589333/manta.log) 

Originally posted by @ajdesalvio in #85

[pontushojer]

Hi @ajdesalvio! Could you check which specific manta and python build was used in manta environment? You could for example do:

ls -l /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75/conda-meta

Alternatively, since there is a fix merged (bioconda/bioconda-recipes#53424) you could try and remove the existing manta conda env (i.e. rm -rf /scratch/user/ajd/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75*) to force snakemake to regenerate it with the latest manta build.

[ajdesalvio]

manta-1.6.0-py27h9948957_4.json

Hi @pontushojer, thank you so much for the quick reply! It looks like the manta version was 1.6.0 and from what I can tell, the python version is 2.7. Here's what the manta.yml file says inside the BLR conda virtual environment:

name: manta
channels:
  - conda-forge
  - bioconda
  - nodefaults
dependencies:
  - bioconda::manta=1.6.0
  - bioconda::samtools=1.12
  - conda-forge::python<3

The temporary environment within the .smakemake folder of the BLR Outputs directory (6437933bd16a31cc5b317d5462f11a75) had a manta .json file (attached) that seemed to confirm what was saved in the .yml. If I remove the existing manta conda env, do you think I could get away with trying blr run --rerun-incomplete, or would it be more advisable to restart the run?

[pontushojer]

Thanks for proviing the JSON!

The build was indeed py27h9948957_4 i.e. the one relating to bioconda/bioconda-recipes#53419.

Definitely try with blr run --rerun-incomplete after removing the env.

Alternatively, before this you could try blr run --conda-create-envs-only to generate the new manta env. Then you could check the JSON to verify that is the latest manta build, i.e. py27h9948957_5.

[ajdesalvio]

Hi @pontushojer, you're welcome and thanks for the suggestions! Just to document everything I've done so far:
I tried blr run --conda-create-envs-only first, and this didn't result in an updated JSON (it was still showing the py27h9948957_4 build).

Next I navigated to the temporary environment within Outputs/.snakemake/conda and removed the temporary environment from before by running rm -rf 6437933bd16a31cc5b317d5462f11a75. With the temporary environment deleted, I reinitialized BLR from the previous run by the following (including this in case I have done something wrong here)

... SLURM parameters ...
module load Anaconda3/2023.07-2 WebProxy GCC/7.3.0-2.30 OpenMPI/3.1.1 git/2.21.0-nodocs Mamba/23.1.0-4 R/3.5.1-bare
export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8
source activate blr_ktz1_v5
ANALYSIS_DIR='/scratch/user/aarondesalvio/Serina/TellSeq2/Outputs_V3/KTZ_1_V5/Outputs'
cd $ANALYSIS_DIR
blr run --rerun-incomplete

The job is still running, but so far I can verify that the updated build has been installed inside Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75/conda-meta. The JSON (attached again) now shows py27h9948957_5.

I'll reply again here with the results of the job when it finishes.

manta-1.6.0-py27h9948957_5.json

[pontushojer]

The job is still running, but so far I can verify that the updated build has been installed inside Outputs/.snakemake/conda/6437933bd16a31cc5b317d5462f11a75/conda-meta. The JSON (attached again) now shows py27h9948957_5.

I'll reply again here with the results of the job when it finishes.

manta-1.6.0-py27h9948957_5.json

Awesome! Let's hope this does the trick 🤞

[ajdesalvio]

I'll reply again here with the results of the job when it finishes.
manta-1.6.0-py27h9948957_5.json

Awesome! Let's hope this does the trick 🤞

Alright, back with an update! We're making a little progress each time. The blr run --rerun-incomplete attempt failed primarily due to the extract_barcodes step failing to find mtglink.gfa.

Next, looking specifically within the manta.log file, the only line in that file was the following: /bin/bash: convertInversion.py: command not found

I tried to locate the convertInversion.py script in the conda environment I'm using and did not find it. Manta is also not found in the conda env (conda list -n blr_ktz1_v5 | grep manta returned a blank line). The YAML inside the temporary environment (6437933bd16a31cc5b317d5462f11a75) however still shows the correct manta build (py27h9948957_5). At this point, do you think restarting from a previous step that was supposed to produce mtglink.gfa would be a good idea (if that is possible), or potentially a fresh installation of BLR using the latest commit and restarting the pipeline?

The log files are not too big so I'm attaching both here. Thanks again for your continued help!

tellseq_ktz1.err..txt
manta.log

[pontushojer]

Seems like they missed to include the convertInversion.py script in the latest builds, I created a new issue for this. Until this is fixed you have to limit to an earlier build, try out branch #89 and see if that works.

[ajdesalvio]

Thanks again @pontushojer, I'll give it a shot and get back to you with results!

[pontushojer]

FYI bioconda/bioconda-recipes#53640 just merged which adds the script to the latest manta build on bioconda.

[ajdesalvio]

That's great news, thank you! I'll try a --rerun-incomplete after deleting the temporary conda env again inside the Outputs directory and I'll also start an entirely new run with a fresh installation of BLR. I will let you know how it goes.

[ajdesalvio]

Hi @pontushojer, I'm back with great news! A successful run finished early this morning! I checked the log last night and saw that snakemake had made it past the manta step so I was hopeful. One thing I should mention is that I completed this run using a BLR clone of fix-88 and created the conda virtual env on 2025-02-03, which I believe predates the manta merge with bioconda that you referenced (this one). Specifically I used:

cd /scratch/user/aarondesalvio/BLR
git clone -b fix-88 https://github.com/AfshinLab/BLR.git BLR_KTZ_1_V6
cd BLR_KTZ_1_V6

conda create --name blr_ktz1_v6 --file environment.linux-64.lock
source activate blr_ktz1_v6
pip install --use-feature=in-tree-build .

Please let me know if I can provide any other details from the run that would be useful for you. I am happy to share results from the MultiQC too if needed. Thanks for everything along the way! I have a few more samples to process and will reach back out if I run into anything else, but hopefully that won't be the case!

[pontushojer]

@ajdesalvio 🥳 That is great to hear! Let's hope the rest of the samples don't give you any more issues 🤞

Since everything was completed successfully with #89 it is good to stick to this manta build for now.