Out of memory despite 990 GB of RAM

[dlaehnemann]

I have tried running scVILP on a dataset of 32 single cells. The data is on from [SRA Bioproject PRJNA168068](https://www.ncbi.nlm.nih.gov/Traces/study/? +++acc=PRJNA168068&f=sample_name_s%3An%3Atnbc%3Blibraryselection_s%3An%3Amda%2Cpcr%3Ac&o=acc_s%3Aa&s=SRR1163035,SRR1163012,SRR1163013,SRR1163019,SRR1163026,SRR1163027,SRR1163034, +++SRR1163043,SRR1163053,SRR1163070,SRR1163074,SRR1163083,SRR1163084,SRR1163091,SRR1163095,SRR1163148,SRR1163149,SRR1163150,SRR1163151,SRR1163152,SRR1163153,SRR1163154,SRR1163155, +++SRR1163156,SRR1163157,SRR1163158,SRR1163159,SRR1163160,SRR1163161,SRR1163162,SRR1163163,SRR1163164,SRR1298936,SRR1163508#) and this publication:
Wang, Y., Waters, J., Leung, M.L., Unruh, A., Roh, W., Shi, X., Chen, K., Scheet, P., Vattathil, S., Liang, H., et al. (2014). Clonal evolution in breast cancer revealed by single nucleus genome sequencing. Nature 512, 155–160 . doi:10.1038/nature13600.

I get an mpileup file of size 3.1 GB. scVILP_main.py with 990 GB RAM available runs for around 14 hours and then crashes with gurobipy.GurobiError: Out of memory. Full scVILP output below. Do you have any recommendations on how I could get scVILP to run on this dataset? Or is it simply too big?

# of taxa: 32
# of mutations: 1056808
false positive rate given: 0.000000
false negative rate given: 0.100000
Using license file /path/to/gurobi/license/file/gurobi.lic
Academic license - for non-commercial use only
Add variables to the model
Traceback (most recent call last):
  File "../scripts/scVILP/scripts/scVILP_main.py", line 242, in <module>
    mat_ = optimize(read_count_mat=read_counts,fp=fp_given,fn=fn_given,missing_data_thr=missing_data_threshold, K_vios=K_, mu0=1e-3, mu1=0.5)
  File "../scripts/scVILP/scripts/scVILP_main.py", line 109, in optimize
    B["("+str(p)+","+str(q)+","+str(k+1)+")"]=model.addVar(vtype=GRB.BINARY, name="B["+str(p)+","+str(q)+","+str(k+1)+"]")
  File "model.pxi", line 2584, in gurobipy.Model.addVar
gurobipy.GurobiError: Out of memory

Mpileup command for creating the input was:

samtools mpileup -f {input.ref} -B -d 1000 -q 40 -Q 30 {input.bams} > {output}

Filtering command was:

python ../scripts/scVILP/scripts/loci_identify.py -in {input.mpileup} -out {output} -names {input.filenames} -ms 2 -nmc 3 

Main command was:

python ../scripts/scVILP/scripts/scVILP_main.py -in {input.filtered_mpileup} -out {params.out_dir} -names {input.filenames}

[dlaehnemann]

I have now seen the exact same error on another dataset, this time only two cells (but whole genome sequencing data). On this dataset, I adjusted the filtering command to:

python ../scripts/scVILP/scripts/loci_identify.py -in {input.mpileup} -out {output} -names {input.filenames} -ms 1 -nmc 1

And the main command here included --mdthr 1.

The gurobipy Out of memory is triggered at a slightly different place:

# of taxa: 2
# of mutations: 31078833
false positive rate given: 0.000000
false negative rate given: 0.100000
Using license file /path/to/gurobi/license/gurobi.lic
Academic license - for non-commercial use only
Add variables to the model
Traceback (most recent call last):
  File "../scripts/scVILP/scripts/scVILP_main.py", line 242, in <module>
    mat_ = optimize(read_count_mat=read_counts,fp=fp_given,fn=fn_given,missing_data_thr=missing_data_threshold, K_vios=K_, mu0=1e-3, mu1=0.5)
  File "../scripts/scVILP/scripts/scVILP_main.py", line 106, in optimize
    V["("+str(p)+","+str(q)+")"]=model.addVar(vtype=GRB.BINARY)
  File "model.pxi", line 2584, in gurobipy.Model.addVar
gurobipy.GurobiError: Out of memory

One peculiar thing that I witnessed in this case was that the Out of memory error was triggered when the process was using only using about 365 GB of RAM with a total of 1050 GB RAM available and the general scVILP_main.py process ran on for at least 5 hours after the error was thrown (without any further logging output).

[dlaehnemann]

Same with a third dataset with 5 single cells and whole exome sequencing data. The total mpileup file size in this case is at only 108 MB. Command line calls as with the first example posted above, but with 1200 GB RAM available and throwing the out of memory error at about 424 GB RAM used, and with the process running for another 8 hours after I noticed the error reported in the output.

# of taxa: 5
# of mutations: 251024
false positive rate given: 0.000000
false negative rate given: 0.100000
Using license file /path/to/gurobi/license/gurobi.lic
Academic license - for non-commercial use only
Add variables to the model
Traceback (most recent call last):
  File "../scripts/scVILP/scripts/scVILP_main.py", line 242, in <module>
    mat_ = optimize(read_count_mat=read_counts,fp=fp_given,fn=fn_given,missing_data_thr=missing_data_threshold, K_vios=K_, mu0=1e-3, mu1=0.5)
  File "../scripts/scVILP/scripts/scVILP_main.py", line 109, in optimize
    B["("+str(p)+","+str(q)+","+str(k+1)+")"]=model.addVar(vtype=GRB.BINARY, name="B["+str(p)+","+str(q)+","+str(k+1)+"]")
  File "model.pxi", line 2584, in gurobipy.Model.addVar
gurobipy.GurobiError: Out of memory