Segmentation fault (core dumped)

[nordhuang]

./squeakr-count -f -k 28 -s 20 -t 1 -o ./ S008_20180206001-8_ffpedna_pan-cancer-v1_5717_S8_R2_001.fq
Reading from the fastq file and inserting in the QF
Segmentation fault (core dumped)

head -8 S008_20180206001-8_ffpedna_pan-cancer-v1_5717_S8_R1_001.fq
@NB551106:74:HG7CWBGX5:2:11106:12634:1554 1:N:0:AGTTCC
ACTCTGGCCTGGGTGACAGAGTGAGACTCGGGCTAAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACAAAAAATAA
+
AAAAAAEEE////A/EAE/E///E6EA/A///<<EE//EEEEEEEEEEEEEEEEE6EEEEEEEEE6EEEE//AE/E<///<<EAAEA/EAAEE6/EEEEEEEEAA<AEEE//E///<<E<<///</E/E///A///A<<///////////
@NB551106:74:HG7CWBGX5:4:22601:22501:19465 1:N:0:AGTTCC
ACTCTGGCCTGGGTGACAGAGTGAGACTCGGGCTAAGAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+
AAAAA/AAE///AA/EAEAE///EAAAA<//<A/EE6EEEEEEEEEEEEEEEEEEEEEEEEEEEAEEEE//<EAA//////<<AA///A//E//E/EEEEE/E<//E///A///////6////</A////////A///6///6///////

I provide fastq files()S008_20180206001-8_ffpedna_pan-cancer-v1_5717_S8_R1_001.fq. Why did it throw a segmentation fault?

[compbio]

I got same problem. Any solution?

test_1.txt
test_2.txt

[rtjohnso]

Can you send me the fastq file that caused the segfault? You can either attach it to your issue report, or send it via email. If it's large, you might try deleting lines to cut it down to a "minimal working example", i.e. a shorter file that still causes squeakr to crash.

Best,
Rob

[compbio]

I loaded first 12 lines of fastq files

[chelseaju]

I am experiencing the same issue. It gives Segmentation fault (core dumped) when processing a fastq file with around 10 million of reads. When I tried to run a smaller file (including the sample file provided - test.fastq), it says

Error opening file for serializing
: No such file or directory 

any idea?

[prashantpandey]

Hi @chelseaju, which command are you running to count k-mers from the sample fastq file (test.fastq)? Because I am not able to reproduce the issue.
I am using this command ./squeakr-count -f -k 28 -s 20 -t 1 -o ./ test.fastq.

Thanks,
Prashant

[prashantpandey]

Hi @nordhuang, I tried running squeakr-count using the fastq file you provided. But I am not getting any segmentation fault. I am using this command ./squeakr-count -f -k 28 -s 20 -t 1 -o ./ tmp.fastq.

Could you please confirm that you are using the same command?

Thanks,
Prashant

[chelseaju]

Hi @prashantpandey , thanks for the quick response. I ran the command line you suggested and it resolved the issue of "No such file or directory". Apparently, the error message rises when the output directory does not exist. However, the same command line still produces "Segmentation Fault" when processing a large number of reads (in my case, more than 387412 lines).

[prashantpandey]

Hi @chelseaju , is there a way I can access your fastq file to reproduce the issue?

Thanks,
Prashant

[chelseaju]

I am attaching the smaller fastq file (with 387416 lines). I also tried line-by-line debugging. It seemed to me that the error occurs at the qf_serialize() function in threadsafe-gqf/gqf.c (line 2139). Unfortunately, I don't really know how to fix this.

test.fq.gz

[prashantpandey]

Hi @chelseaju, just wanted to check you are seeing the seg fault issue with the file (smaller fastq) you uploaded?
Also, could you specify the exact comment you are using?

Thanks,
Prashant

[chelseaju]

@prashantpandey I used the command you suggested
./squeakr-count -f -k 28 -s 20 -t 1 -o . test.fq

[prashantpandey]

Hi @chelseaju,

I tried reproducing your bug but I am actually able to count k-mer in test.fq file that you provided without any bug.

./squeakr-count -f -k 28 -s 20 -t 1 -o . test.fq
Reading from the fastq file and inserting in the QF
Total Time Elapsed: 3.161622seconds
Calc freq distribution: 
Total Time Elapsed: 0.020426seconds
Maximum freq: 129
Num distinct elem: 732988
Total num elems: 4643908

[accopeland]

Hi,
I'm also seeing segfaults. I'm attaching the smallest file that produces an error on my machine (4316 pairs). Machine is

Linux dint01 2.6.32-696.18.7.el6.nersc.x86_64 #1 SMP
 product: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
       vendor: Intel Corp.
       physical id: 1
       bus info: cpu@0
       size: 2601MHz
       capacity: 2601MHz
       width: 64 bits
       capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid cpufreq

Compiled with NH=1 (otherwise see illegal instruction error).

Commands producing segfault (any 13<=k<=29) with uncompressed or gzip fastq. Strangely, k=11 works as do k=29 and various values up to 61, but I did not test exhaustively. Thread count doesn't seem to matter.

squeakr-count -f -k 13 -s 20 -t 22 -o ./ x.fq
Reading from the fastq file and inserting in the QF
Segmentation fault

x.fq.gz

[Christina-hshi]

Found a bug that causes segmentation fault of the program.
Hi all,
When running the program with parameters given as the example in README file.
./squeakr-count -f -k 28 -s 20 -t 1 -o .
I also got the segmentation fault.
Then I used GDB to see where went wrong inside the code. I found that the codes near line 1360 in "gqf.c" are not safe.
`
1360 uint64_t empty_slot_index = find_first_empty_slot(qf, runend_index+1);

1362 shift_remainders(qf, insert_index, empty_slot_index);
`
In line 1360, it tried to find out the first_empty_slot at or after slot (runend_index+1). However, there could be no empty slot after it, so the returned slot index is larger than the total number of slots, which is not a valid index. As the result, it leads to a segmentation fault when accessing memory not owned by it in line 1362.
So here we need to check whether "empty_slot_index" is out of bound.

What's more important, I think the real reason which causes the problem is the unreasonable parameters.
For example, if we set "-k 28 -s 20", then we know the maximum number of different K-mers is 2^(2*28)=2^56 and the number of slots in RSQF is ~2^20. If we assume all objects stored in RSQF has frequency at least 3 and the hash_bits is 20(should be >=20 based on parameter '-s'), then based on the encoding scheme of the RSQF, it uses at least 3 slots to store each object. So actually, the RSQF can store at most (2^20)/3 objects. It is possible that after hashing, we get 2^20 different objects because 2^56 is larger than 2^20. That means RSQF will run out of empty slots after inserting (2^20)/3 unique objects. As a result, the "segmentation fault" will happen.
Actually in the program, the hash bits is set to be s+8, so there is even higher chance to get "segmentation fault"(run out of owned space), since we may need to store 2^28 unique objects in memory blocks that can only holds at most (2^20)/3 unique objects with frequency >=3.

How to find reasonable parameters?
The key step to find reasonable parameters is to estimate the number of distinct objects that are going to be inserted into RSQF.
In theory, the maximum number of distinct objects is min(4^k, 2^hash_bits) if we don't consider the amount of data we use and its specific properties. If 2^hash_bits <= 4^k, then since the RSQF can only store at most (2^hash_bits)/3 objects under our assumption, so it is very likely to run out of space and lead to potential errors. Thus, the solution seems to set k such that 4^k <= (2^hash_bits)/3. For example, if hash_bits is 20, then the k should be <=4. If we want to use large k, then we need increase hash_bits accordingly. For example, if we want k=28, then the hash_bits should be >=58. If each slot uses only 1 byte, then we need 2^58 bytes, which is memory prohibitive.
Fortunately, the number of distinct objects is much smaller than the maximum number in theory in many real cases. For example, we want to build the K-mer spectrum for human genome using RSQF. Since the human genome is around 3 billions bp long, so there will be at most ~2*3=6 billion(by considering also reverse complement strand) unique K-mers if we assume no or extremely low sequencing errors in our data. So no matter how big the K is, the number of unique K-mers will always be bounded by ~6 billions. Therefore, we can set hash_bits >= 35 such that we will have low chance to run out of slots.

[prashantpandey]

Hi @Christina-hsh Thanks for looking into the segfault. You are right that the segfault happens because the number of slots in the CQF (counting quotient filter) is not enough.
The example command ./squeakr-count -f -k 28 -s 20 -t 1 -o . is to count k-mers in the test.fastq file which contains fewer than 2^20 28-mers.

However, for other fastq files, to decide the correct size of the CQF (or -s argument) we have a script. This script takes as input the fastq file(s) and estimates the log of the number of slots needed for the Squeakr. Please try this script. It is also mentioned in the README.
lognumslots.sh script can be used to estimate the log of number of slots in the CQF argument. The script takes as input the path to the output file of 'ntCard' (https://github.com/bcgsc/ntCard). It then calculates log of the number of slots needed by Squeakr to count k-mers.

Thanks,
Prashant