parallelize

[maxcw]

It would be great if there's an option for embarrassingly parallel computations, especially if all N^2 distances are calculated.

[vc1492a]

@maxcw Thanks for opening the issue, I agree that it would be a nice option to provide parallelism as part of the available options for computation. I believe this is available via numba, the JIT-compilation library that's an option when using PyNomaly.

Since parallel computation is an option when using numba, it may be pretty straight-forward to try and test the following implementation, more specifically take this line:

PyNomaly/PyNomaly/loop.py

Line 560 in 744fa57

compute = numba.jit(self._compute_distance_and_neighbor_matrix,

In pass the parameter parallel in the following way:

# parallel is some boolean parameter set earlier, e.g. 
parallel = True
compute = numba.jit(self._compute_distance_and_neighbor_matrix,
                            cache=True, parallel=parallel) if self.use_numba else \
            self._compute_distance_and_neighbor_matrix

I'll mark this as an enhancement to take a look at for a future release (or please feel free to try it yourself and submit a PR). Thanks!

[vc1492a]

Work on this issue can now be tracked in #43.

[vc1492a]

May be helpful to use a tracing tool like pyinstrument to gauge the effect of certain code changes.

[vc1492a]

Implemented in the branch feature/numba parallel but performance is not improved.