It appears that there is likely too much disk thrashing when randomly selecting audio sources for impulse responses and/or noise. Cache these files in memory (when footprint allows) to ensure that this can be computed faster.
Alternatively look at alternative forms for storing the corpus (rather than reading in raw wavs every time).