Issue with loading trusted k-mers

[standage]

Hi @mourisl,

It was exciting to meet you at CSHL and I appreciate that you took some time to implement the feature we discussed.

I gave it a try this morning, and saving the trusted k-mers file seemed to work fine. At least no error was reported.

$ time lighter \
    -r mother-reads-1.fq -r mother-reads-2.fq \
    -r father-reads-1.fq -r father-reads-2.fq \
    -r proband-reads-1.fq -r proband-reads-2.fq \
    -saveTrustedKmers trusted-kmers \
    -k 25 25000000 0.233 -t 4
[2017-11-06 10:54:06] =============Start====================
[2017-11-06 10:54:07] Bad quality threshold is "7"
[2017-11-06 10:54:54] Finish sampling kmers
[2017-11-06 10:54:54] Bloom filter A's false positive rate: 0.162178
[2017-11-06 10:54:54] The error rate is high. Lighter adjusts -maxcor to 5 and bad quality threshold to "8".
[2017-11-06 10:56:30] Finish storing trusted kmers
[2017-11-06 10:56:30] The trusted kmers are saved in file trusted-kmers.

real	2m23.730s
user	7m45.252s
sys	0m12.727s

There was a problem when I tried to run the error correction though.

$ time lighter -r mother-reads-1.fq -r mother-reads-2.fq  -k 25 25000000 0.233 -t 4 -loadTrustedKmers trusted-kmers -od .
[2017-11-06 11:07:27] =============Start====================
[2017-11-06 11:07:28] Bad quality threshold is "7"
[2017-11-06 11:07:28] Finish loading trusted kmers from file trusted-kmers.
Segmentation fault: 11

real	0m1.548s
user	0m1.385s
sys	0m0.288s

I figured I would try without threading, which did prevent the memory handling error, but did not find any errors to correct.

$ time lighter -r mother-reads-1.fq -r mother-reads-2.fq  -k 25 25000000 0.233 -loadTrustedKmers trusted-kmers -od .
[2017-11-06 11:07:52] =============Start====================
[2017-11-06 11:07:53] Bad quality threshold is "7"
[2017-11-06 11:07:53] Finish loading trusted kmers from file trusted-kmers.
[2017-11-06 11:17:43] Finish error correction
Processed 6000000 reads:
	0 are error-free
	Corrected 0 bases(0.000000 corrections for reads with errors)
	Trimmed 0 reads with average trimmed bases 0.000000
	Discard 0 reads

real	9m50.509s
user	9m42.973s
sys	0m6.276s

Is it possible I'm invoking the software incorrectly?

---

I gzipped this data set and uploaded to AWS, in case that helps to reproduce the error. Please let me know when you've downloaded the data files so I can revoke public access.

• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/mother-reads-1.fq.gz
• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/mother-reads-2.fq.gz
• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/father-reads-1.fq.gz
• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/father-reads-2.fq.gz
• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/proband-reads-1.fq.gz
• https://s3.amazonaws.com/nanotrio-lighter-reads-temp/proband-reads-2.fq.gz

[mourisl]

Thanks for sharing the data with me. I've tested it on two systems and it all works for the multi-threading. Have you tried to "make clean" and recompile the program?

For the 0 correction, after testing, I believe it is due to the high error rate of bloom filter A and Lighter would regard every kmer as weak then. I think because you've used 3 data sets to obtain solid kmers, the actual read coverage is tripled as a result and the sampling rate should be 0.233/3, which is about 0.078.

[standage]

Heh, I should have noticed the warning message in my first command that said the error rate was too high. Running the two commands with the adjusted error rate seems to have fixed the problem.

However, I'm still getting a segfault when running the second command with -t 4. I'm on a Mac laptop. I'll see if I have the problem on another machine.

[standage]

Curious. The threading caused a segfault on another Mac machine, but the commands worked just fine on a Fedora Linux machine. I wonder if it's compiler related: clang vs gcc? I certainly got a lot of compiler warnings with clang on Mac but none with g++ on Linux.

[mourisl]

I only tested it on gcc and some linux systems. I don't have a Mac so I'm not sure how things work there.

[standage]

When I compile with clang on Linux, I get the same segfault error. There is probably some undefined/unguaranteed behavior in the code that gcc fixes automatically but clang does not. But in any case, I will make sure to use gcc on Linux for this feature. Thanks!