samtools sort: fail to open "./samtools.36604.8950.tmp.1019.bam": Too many open files

[sanjin725]

Dear administrator.

Hello, this error occurred when I was using juicer to process HIC data. How should I solve this problem?
The error is:
[bam_sort_core] merging from 1530 files and 170 in-memory blocks...
[E::hts_open_format] Failed to open file "./samtools.214728.6262.tmp.1019.bam" : Too many open files
samtools sort: fail to open "./samtools.214728.6262.tmp.1019.bam": Too many open files
***! Failure during chimera handling of /gpfs/home/liyixian/soft/3DDNA/juicer/splits/hic.fastq.gz
I also provided a complete log below:
juicer.log
Good luck, looking forward to your reply
Xiaoxin Pei

[yuan102379]

I have also meet with the simmilar issue? Have this comment been solved ?

[BRNiu]

Same issue. Is there any solution?

[cohen-r]

Hi! I am having the same issue. Has anyone figured out a solution?

[ycl6]

Hi all, I have submitted a PR (#362) which hopefully will address this issue.

When running juicer (from CPU/juicer.sh), you can set the memory per thread used by samtools sort with a new --mem flag. The default memory used by samtools sort is 768M per thread. By setting a larger memory per thread, more data can be stored in memory before the content is written to a temporary file, therefore reducing the total number of opened temporary files required during the sort process.

Please take note of the total amount of memory that is available on your processing platform and the number of threads you are using. For example, if there are 64G memory available to you and you are running with 8 threads. Then the maximum memory per thread should be 8GB (i.e. -t 8 --mem 8G).