error at step5

[MichelMoser]

Dear Jon,

I reached step5 but get a core dump.

Sterr:

/var/spool/slurmd/job08977/slurm_script: line 19: 103982 Aborted                 (core dumped) /home/mmoser/w2rap-contigger/bin/w2rap-contigger -t 8 -m 100 -r /data/users/mmoser/Illumina/Peaxi_pe01kb_UIL2013_P1.fq,/data/users/mmoser/Illumina/Peaxi_pe01kb_UIL2013_P2.fq -o contigs -p pax_k200 --disk_batches 20    

By looking at stout, it is unclear to me if some parameter setting might resolve the problem.

Stout:

--== Step 5: Assembling gaps ==--
Fri Apr 20 22:58:35 2018: inverting paths
Fri Apr 20 22:59:05 2018: Finding unsatisfied path clusters
Fri Apr 20 23:00:05 2018: Merging 8909194 clusters
Fri Apr 20 23:10:28 2018: 1192450 non-inverted clusters
Fri Apr 20 23:13:36 2018: processing 1192450 blobs
ForceAssertGt(MemAvailable(),0ul) at /home/mmoser/w2rap-contigger/src/kmers/naif_kmer/NaifKmerizer.h:560 failed in function
void naif_kmerize(KERNEL_t*, size_t, size_t, size_t) [with KERNEL_t = PreCorrector<Kmer29H, MasterVec<SerfVec<unsigned char> > >; size_t = long unsigned int]
with values arg1 = 0 and arg2 = 0
ForceAssertGt(MemAvailable(),0ul) at /home/mmoser/w2rap-contigger/src/kmers/naif_kmer/NaifKmerizer.h:560 failed in function
void naif_kmerize(KERNEL_t*, size_t, size_t, size_t) [with KERNEL_t = PreCorrector<Kmer29H, MasterVec<SerfVec<unsigned char> > >; size_t = long unsigned int]
with values arg1 = 0 and arg2 = 0

Additionally, when looking into the assembly-folder, i got an pax_k200.first.frags.dist.png.FAIL -file telling me some execs are missing on the cluster i run w2rap:

Could not generate frags.dist.png because the following executables were not found:      
pstopnm,pnmcrop,pnmpad,pnmtopng.

But i assume this would not cause the assembly to terminate, right?

Could you help me to get arround this error?

Thank you,
Michel

[jonwright99]

Hi Michel,

You're right, the pax_k200.first.frags.dist.png.FAIL file is nothing to do with the error. It looks like it might be something to do with your data. Could you send me your read files so we can try to replicate the error and let me know which version of w2rap you are running? There should be a git commit flag in the log file.

Thanks,
Jon

[jonwright99]

It could also be a out-of-memory error as step 5 is the other memory intensive step. Are you able to increase the memory for this step or sub-sample your reads and run again from step 1?

[MichelMoser]

your right, might be another memory-issue. an assembly with 1/5 of the data ran without problems.
could you tell the estimated memory requirement for a 1 Gb genome and 60x coverage?
I can send you the reads or a subset of it via dropbox.
Thank you

[jonwright99]

Memory requirements are difficult to estimate as they will depend on the quality of your sequencing and the heterozygosity of your genome. We usually plot a kmer frequency distribution from the reads before assembly to check this - see https://github.com/bioinfologics/w2rap/

[Wenfei-Xian]

Hi Michel, Jon
Have you solved the problem? I also meet the problem . "error in step5"

"
memory in use now = 108992524288
Fri Sep 21 00:35:53 2018: back from buildBigKHBVFromReads
3.27 minutes used in new stuff 2 test
peak mem usage = 222.07 GB
52.3 seconds used in new stuff 5
Fri Sep 21 00:37:00 2018: finding interesting reads
Fri Sep 21 00:37:13 2018: building dictionary
Fri Sep 21 00:38:25 2018: reducing

Fatal error (pid=5865) at Fri Sep 21 00:38:25 2018:
Insufficient memory.
"

It show the peak men usage =222.07GB
but My server has 3T of memory ，There should be no memory shortage.
Also ,When I reduce the reads data to 1/5 ,it worked well. For the insufficient data ,I couldn't get the complete genome ,,,,

Thank you

[kokyriakidis]

Hi @jonwright99

I get the same error at stage 5:

Mon Jan 28 21:43:13 2019: building hb2
12.2 seconds used in new stuff 1 test
memory in use now = 244648316928
Mon Jan 28 21:48:48 2019: back from buildBigKHBVFromReads
5.61 minutes used in new stuff 2 test
peak mem usage = 333.26 GB
36.7 seconds used in new stuff 5
Mon Jan 28 21:50:15 2019: finding interesting reads
Mon Jan 28 21:50:45 2019: building dictionary
Mon Jan 28 21:53:20 2019: reducing
We need 4 passes.
Expect 383574 keys per batch.
Provide 598953 keys per batch.
Tue Jan 29 03:59:55 2019: kmerizing

Fatal error (pid=39910) at Tue Jan 29 03:59:55 2019:
Insufficient memory.

And my node has 500gb of RAM. Any suggestions?

[jonwright99]

Can you post the command line you are using to start the contigger please? There is a soft memory limit parameter that you might not be including (-m) which defaults to 10Tb.

[kokyriakidis]

@jonwright99

w2rap-contigger/bin/w2rap-contigger -t 64 -r /mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519283_R1.fastq,/mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519283_R2.fastq,/mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519284_R1.fastq,/mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519284_R2.fastq,/mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519285_R1.fastq,/mnt/pharmngs_a/home/kyriakidk/w2rap_Project/data/ERR519285_R2.fastq -o contigs -p kiwi3_K200 --min_freq 9 -d 16 --dump_all 1 &>> conlog.log

Yes I didn't use -m. If I use -m will it solve the problem? In a node with 500GB, what do I have to use in -m? -m 500GB ??

[jonwright99]

It's a soft limit so I usually set it to 4/5ths of available memory which in your case will be 400. You could try setting this. It definitely looks like a memory issue, the log for the previous part is using 333Gb so it may well be requiring more than 500Gb. You could also try reducing the number of reads you use as input. If this is a memory issue, there's no way around it, you'll need more memory to assemble the coverage you have.