Memory issue when subsample catalogs

[echaussidon]

The following code should raise a memory issue:

import numpy as np
import mpytools as mpy
from mpi4py import MPI
mpicomm = MPI.COMM_WORLD

# read a catalog on several ranks
cat = mpy.Catalog.read('one_catalog.fits', columns=['X', 'Y'], mpicomm=mpicomm)
# select a sub part of the catalog
sel = np.random.rand(cat.size) <= 0.5
cat = cat[sel]

print(cat['Y'])  # --> create a memory error

A dirty fix is to ready before the selection all the column that you need after the subselection:

import numpy as np
import mpytools as mpy
from mpi4py import MPI
mpicomm = MPI.COMM_WORLD

# read a catalog on several ranks
cat = mpy.Catalog.read('one_catalog.fits', columns=['X', 'Y'], mpicomm=mpicomm)
_, _ = cat['X'], cat['Y']
# select a sub part of the catalog
sel = np.random.rand(cat.size) <= 0.5
cat = cat[sel]

print(cat['Y'])  # --> create a memory error