CUDA out of memory

[antss8]

CUDA out of memory. Tried to allocate 14.03 GiB. GPU 0 has a total capacity of 79.14 GiB of which 8.49 GiB is free. Process 17884 has 70.64 GiB memory in use. Of the allocated memory 70.14 GiB is allocated by PyTorch, and 18.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

After I entered a 16000 * 61000 matrix and rd_trainer.train() , I get these error messages.
Does RegDiffusion need such a large amount of memory to support training? The A800 graphics card we used already has a large enough.
Is it possible to do distributed arithmetic to prevent large amounts of data from being loaded into the memory?

Here is my code:
import numpy as np
import pandas as pd
import scanpy as sc
import regdiffusion as rd
import loompy as lp
from pyscenic.rss import regulon_specificity_scores
from pyscenic.plotting import plot_rss
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_theme(style="white")

wd = './'
data_path = f'{wd}/subT.loom'
cisdb_path = f'{wd}/TF-motif.DB/Gh_cistarget_database/Gh.regions_vs_motifs.rankings.feather'

adata = sc.read_loom(data_path, sparse=True)
cisdb = pd.read_feather(cisdb_path)
adata = adata[:, adata.var_names.isin(cisdb.columns)]
x = adata.X.toarray()
rd_trainer = rd.RegDiffusionTrainer(x)
rd_trainer.train()

[haozhu233]

Can you clarify if 61000 is the dimension of genes or cells? Yes, RegDiffusion does requires large memory for training. With 80G memory, it’s sufficient to analyze all of the 20k protein coding genes (I think 20k takes about 40G) but it won’t be enough for the entire 60k genes (with promoter, enhancer etc). In this case, you will have to do some filters on genes (remove low expressed genes or only use highly variable genes). It’s certainly a limitation that we should have mentioned in the paper. It is certainly possible to do distributed arithmetic but we have not implemented it.

…

On Wed, Jun 25, 2025 at 06:24 antss8 ***@***.***> wrote: *antss8* created an issue (TuftsBCB/RegDiffusion#12) <#12> CUDA out of memory. Tried to allocate 14.03 GiB. GPU 0 has a total capacity of 79.14 GiB of which 8.49 GiB is free. Process 17884 has 70.64 GiB memory in use. Of the allocated memory 70.14 GiB is allocated by PyTorch, and 18.26 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management ( https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) After I entered a 16000 * 61000 matrix and rd_trainer.train() , I get these error messages. Does RegDiffusion need such a large amount of memory to support training? The A800 graphics card we used already has a large enough. Is it possible to do distributed arithmetic to prevent large amounts of data from being loaded into the memory? Here is my code: import numpy as np import pandas as pd import scanpy as sc import regdiffusion as rd import loompy as lp from pyscenic.rss import regulon_specificity_scores from pyscenic.plotting import plot_rss import matplotlib.pyplot as plt import seaborn as sns sns.set_theme(style="white") wd = './' data_path = f'{wd}/subT.loom' cisdb_path = f'{wd}/TF-motif.DB/Gh_cistarget_database/Gh.regions_vs_motifs.rankings.feather' adata = sc.read_loom(data_path, sparse=True) cisdb = pd.read_feather(cisdb_path) adata = adata[:, adata.var_names.isin(cisdb.columns)] x = adata.X.toarray() rd_trainer = rd.RegDiffusionTrainer(x) rd_trainer.train() — Reply to this email directly, view it on GitHub <#12>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVQRPSBNECAPMOFUWSBZFL3FJ2FZAVCNFSM6AAAAACAC3M2NKVHI2DSMVQWIX3LMV43ASLTON2WKOZTGE3TKMBZHE3TONI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[antss8]

Thank you for your reply. 61000 is the dimension of genes. Do you mean that the amount of memory used is related to the number of genes, not the number of cells?

Also, if I use a loom file as input, can I run RegDiffusion with only the CPU? Is it possible to solve the problem of insufficient memory on the graphics card？The memory of CPU can have 258G or more.

[haozhu233]

Yes. The runtime of regdiffusion does not depend on cell. # of gene is the only driving factor.

Also, yes, you can run it on CPU. It's still pretty fast compared with grnboost but is much slower compared with the GPU version.

One of the ideas in my mind is to optimize the code on CPU learning with some form of parallelization but that requires quite a bit of work.

[antss8]

It's good work. I wonder if it's possible to reduce the amount of data loaded into the graphics card's memory at once by supporting the input of sparse matrices. It would be even better if we could achieve parallelization.

[haozhu233]

Speaking of that, you should be able to reduce the GPU memory consumption by using smaller batch size (the default is 128). However, that means that you also need to adjust the lr and possibly the number of training iterations.

[antss8]

I tried to run it with two or more graphics cards, but regdiffusion will only use one. I tried DDP and configured a distributed training environment, but it always reports an error. Do you have a good idea for this?

ERROR SUCH:
TypeError: RegDiffusion.forward() missing 1 required positional argument: 'ct'
AttributeError: 'RegDiffusion' object has no attribute 'celltype_emb'
and so on.

Also, how are batch size, the lr and the number of training iterations set to adjust. Thanks !

[haozhu233]

To do DDP, you need to at least implement a different trainer. You may consider using my implementation as a reference. https://github.com/TuftsBCB/RegDiffusion/blob/master/regdiffusion/trainer.py.

In fact, it would be great if you are willing to submit a PR and contribute. I'm also happy to discuss a bit in details. my email is haozhu233@gmail.com. Maybe we can get some work done.

[haozhu233]

In terms of adjusting batch size, right now it's at 128. You definitely want to decrease it to reduce the memory load. I would say go down to 16 or 32? I need to test it out to confirm.

In terms of lr, that's a bit more complicated. https://stackoverflow.com/questions/53033556/how-should-the-learning-rate-change-as-the-batch-size-change.

You also want to train the model for longer. If your new batch size is 32, you should train the model for 4k iterations.

[haozhu233]

One thing I'm going to add is to have the option to reduce the batch_size while do gradient accumulation. So we don't need to worry about learning rate at least.