Sparse matrix support

[sina-mansour]

As detailed in the documentation, handling dense structures can be computationally burdensome. Essentially, an increase in the number of nodes may introduce complexities due to the cubic growth of the distance matrix size.

I particularly had a feature request, which might very well be already available but not explained in the documentation. I was wondering if brainsmash can also accept scipy.sparse distance matrices (or possibly inverses of distance such that zero would denote very long distances). This would result in a potentially considerable computational speedup and also reduce storage requirements for the dense distance matrix.

[jbburt]

BrainSMASH does indeed use sparse matrix routines in the workbench sub-package, though I think they're used more for algorithmic efficiency (during graph traversal when deriving the cortical distance matrix) than to mitigate memory burden. The dense matrices are so large that they're never loaded entirely into memory anyways (via memory-mapping). In my analyses, the distance matrix for ~10^5 voxels was ~20 GB. Since I only ever needed to compute it once, the storage burden wasn't considerable. Is your situation different?

[sina-mansour]

Yeah, basically, I was interested in testing the impact of using different distance matrices (e.g. at different levels of sparsity, or coming from different distance/similarity notions) and basically visualize how that changes the nulls produced by the method. Ideally, this could be facilitated by the possibility of loading a sparse matrix denoting the inverse of distance. This way a value of zero would indicate infinite (very long) distances and the same operation may be achieved by sparse matrices which may even take less than a GigaByte of storage/memory.