"Out of memory" error when query compression is not used

[ilnamkang]

I've encountered "Out of memory" error when I analyzed about 10,000 metagenome reads against 'nr-20140917-cablastx' database using cablastp-xsearch.
(The program wan run on Ubuntu server with 128 GB memory and 16 cores.)

And, it seems that the memory error could be avoided by using query compression (by setting --compress-query="true" in command line).

Should I use query compression to analyze a large number of metagenome reads by cablastp-xsearch?

[ndaniels]

Yes, please try using query compression.

On Apr 14, 2015, at 3:38 AM, ilnamkang notifications@github.com wrote:

I've encountered "Out of memory" error when I analyzed about 10,000 metagenome reads against 'nr-20140917-cablastx' database using cablastp-xsearch.
(The program wan run on Ubuntu server with 128 GB memory and 16 cores.)

And, it seems that the memory error could be avoided by using query compression (by setting --compress-query="true" in command line).

Should I use query compression to analyze a large number of metagenome reads by cablastp-xsearch?

—
Reply to this email directly or view it on GitHub.

[ndaniels]

I should note that I am planning to incorporate an iterative query mode even without query-side compression. Right now, if query-side compression is not used, all coarse hits from the coarse search need to be decompressed at once, and searched. But if there's not a lot of overlap among those hits, it's going to end up doing a lot of unnecessary searches (this will depend on the breadth of the queries). With an iterative query mode, I expect better runtime performance for very large queries even when query-side compression is not used (or applicable; query-side compression is only helpful for high-coverage shotgun reads).

[ilnamkang]

It seems that I can avoid the memory problem by using query-side compression and/or iterative query mode according to your suggestion. But, when I used the above options together with output file option, it seems that the output file is overwritten repetitively and frequently.

My command was something like as follows:
./bin/cablastp-xsearch --compress-query="true" (--iterative-queries="true") database query.fasta --blast-args -outfmt 6 -out test.txt

When I performed the above command, the output file "test.txt" was repetitively overwritten and remained with no or very small file size.

How can I avoid this problem?

Thanks for your help.

[ndaniels]

It seems you have found a bug. I never tested passing the -out flag as a blast argument. Instead, try using output redirection: > out.txt

That's what we used for all tests. But I will make a note of this.

On Apr 15, 2015, at 4:25 AM, ilnamkang notifications@github.com wrote:

It seems that I can avoid the memory problem by using query-side compression and/or iterative query mode according to your suggestion. But, when I used the above options together with output file option, it seems that the output file is overwritten repetitively and frequently.

My command was something like as follows:
./bin/cablastp-xsearch --compress-query="true" (--iterative-queries="true") database query.fasta --blast-args -outfmt 6 -out test.txt

When I performed the above command, the output file "test.txt" was repetitively overwritten and remained with no or very small file size.

How can I avoid this problem?

Thanks for your help.

—
Reply to this email directly or view it on GitHub.