Memory corruption

[roemerseb]

When running the map based heterogeneous model with fMRI data from one of my own subjects, the job fails at an early step due to memory issues (see picture). I tried to limit the cmaes size by setting the popsize =5 and n_iter = 5, but the issue stays the same. I tried to run the code both using singularity or directly downloading cubnm in an environment, which didn't change the result. The job failed when hitting 250 GB memory.
I would be very grateful for any ideas how to tackle the issue. Any ideas what could be wrong about my input, that causes this excessive need for memory and eventual fail?

this is the FC matrix I calculated and wanted to feed, as well as the derived FCD
code used:
FC:
emp_fc_tril = utils.calculate_fc(emp_bold_zscored, exc_interhemispheric=True, return_tril=True)

FCD:
emp_fcd_tril= utils.calculate_fcd(
emp_bold,
window_size=window_size,
window_step=window_step,
exc_interhemispheric=True,
return_tril=True
)

FC.csv
FCD.csv

[amnsbr]

@roemerseb and I have been discussing this off GitHub. To summarize our conversation here: this issue is most likely related to the very small window size (2 TRs) that was used, which combined with a long simulation duration, had resulted in a very large dynamic FC and FCD matrix that exceeded the available memory.

I haven’t yet reproduced this error on my end to investigate it further, but a memory leak or corruption seems not very likely to me. One way to prevent such errors could be to define a minimum window size (in TRs) on the Python side and raise an error if the user-indicated window size is smaller than that.

In the long run, a better solution would be to predict on Python side what will the memory requirements of a SimGroup run will be, and raise an error if it exceeds the available GPU/CPU memory (related: #36).