Millions of repeated errors in reads.preprocess.log file

[terrycojones]

Hi. Thanks for putting the SURPI code out, and for making the Amazon AMI.

I'm running the AMI on an i2.4xlarge, as recommended.

Sometime after running SURPI.sh, apparently successfully, I happened to look in reads.preprocess.log and was surprised to see it contained many errors. These are of two types:

cat: xaa.modheader.cutadapt.summary.log: No such file or directory
cat: xaa.modheader.adapterinfo.log: No such file or directory
cat: xaa.cutadapt.cropped.fastq.log: No such file or directory
cat: xaa.preprocessed.fastq: No such file or directory
cat: xaa.cutadapt.cropped.dusted.bad.fastq: No such file or directory
cat: xab.modheader.cutadapt.summary.log: No such file or directory
cat: xab.modheader.adapterinfo.log: No such file or directory
cat: xab.cutadapt.cropped.fastq.log: No such file or directory
cat: xab.preprocessed.fastq: No such file or directory
cat: xab.cutadapt.cropped.dusted.bad.fastq: No such file or directory

Etc. And, following the above:

cat: xaq.preprocessed.fastq: No such file or directory
cat: xaq.cutadapt.cropped.dusted.bad.fastq: No such file or directory
Wed Jun 11 05:39:21 PDT 2014    preprocess_ncores.sh    done concatenating output...
Wed Jun 11 05:39:21 PDT 2014    preprocess_ncores.sh    including duplicates (did not run UNIQ)
Wed Jun 11 05:39:21 PDT 2014
Wed Jun 11 05:39:21 PDT 2014    preprocess_ncores.sh    SPLITTING time: 8 seconds
Wed Jun 11 05:39:31 PDT 2014    preprocess_ncores.sh    median CUTADAPT time per core: 11 seconds
Wed Jun 11 05:39:49 PDT 2014    preprocess_ncores.sh    median DUST time per core: 11 seconds
Wed Jun 11 05:39:49 PDT 2014    preprocess_ncores.sh    TOTAL TIME: 30 seconds
Wed Jun 11 05:39:49 PDT 2014    preprocess_ncores.sh    TOTAL CLOCK TIME (INCLUDING OVERHEAD): 607 seconds
l -h' for more information.
Exit program.

ERROR: Cannot open file : No such file or directory.

Try 'perl prinseq-lite.pl -h' for more information.
Exit program.

ERROR: Cannot open file : No such file or directory.

The file of reads I'm trying to process has 12M sequences in it. I get the above error (#2) in the reads.preprocess.log file 114M times!

Can these errors be safely ignored? Shouldn't they have appeared on stderr instead of silently being put into a log file?

[sfederman]

Hi Terry,

Thanks for the logs - and for your interest.

It is difficult to troubleshoot the issue without knowing a bit more about your input, and no, these issues should not occur in a successful SURPI run.

Can you send us a few things to help us troubleshoot?

• first 10 reads or so of your FASTQ input file
• SURPI log (apparently entitled SURPI.reads.log in your case) generated by the above run
• your config file (reads.config)
• directory listing of the run folder so far (ls -lht)

[terrycojones]

Hi Scot

Thanks for the reply. Details are in http://jon.es/other/terry-surpi.zip

Let me know how/if I can help.

Terry

[terrycojones]

BTW, I can give you a login to the EC2 instance if you'd like to take more of a look around. Just send me an ssh key if you want in. I'm terry@jon.es

The SURPI run is still going, after 12 hours, which surprised me given the figures quoted in http://genome.cshlp.org/content/early/2014/05/16/gr.171934.113.full.pdf I'm wondering if the long run time might have to do with the errors above.

Thanks! :-)

[sfederman]

Thanks for the info - here's what we believe is happening:

You've selected the Illumina quality format for your input file, which counterintuitively may be incorrect if your sequencing was conducted on an Illumina machine after early 2011. (See http://en.wikipedia.org/wiki/FASTQ_format for more information). We will clarify this in our documentation.

Selecting the incorrect quality will lead to the failure you are experiencing where the file reads-numbered.preprocessed.fastq is improperly generated (of 0 size). Currently SURPI will continue, but we are adjusting the pipeline to fail (loudly) when this occurs.

Try adjusting the quality parameter as below (within your config file):

quality="Sanger"

then rerun SURPI, and verify that a non-zero size FASTQ file entitled reads-numbered.preprocessed.fastq is generated.

Thanks, and please do let us know how it works out with your data.

-Scot

[terrycojones]

Hi Scot. Thanks, I changed the quality setting and it all ran cleanly (in 85 mins). I'll close this ticket and maybe send you a pull request to exit if the preprocessed fastq file is empty.