Memory overallocation in process_somaticcaller_high #113
Description
Description of the bug
Hi,
We are writing to report a memory efficiency issue observed in the LOGAN pipeline on Biowulf, and to request an adjustment to the Octopus step's resource allocation.
Observed Issue:
For jobs with input sizes ranging from 29–45 GB (fastq.gz), we observed significantly low memory efficiency:
- Memory Needed: 60,869 GBh
- Memory Allocated: 372,745 GBh
- Efficiency: 16.3%
After investigation, the Octopus step — currently starting at 96 GB memory — appears to be the primary contributor, with a median memory utilization of only 15.7%.
Request:
Since Nextflow supports dynamic resource reallocation (failed jobs are automatically resubmitted with increased memory), the initial 96 GB allocation is unnecessarily high for typical WGS inputs in this size range.
We recommend reducing the starting memory for the Octopus step from 96 GB → 30 GB, which would:
- Better reflect actual memory usage (~15 GB median)
- Significantly improve memory efficiency and cluster resource utilization
- Not impact pipeline reliability, as Nextflow will automatically retry with higher memory if the job fails
Could you please consider updating the resource configuration for the Octopus process?
Thank you for your time.
Qi
Command used and terminal output
Relevant files
No response
System information
No response