Build a Catalog class to deal with everything related to catalog call. The catalog will be handle through dask. This will require to write high-level method to make it easier to use and more transparent for the user.
Using dask will allow to use large catalogs transparently. Also, dask does more than catalog handling. The computation on the catalog are done either on a LocalCluster or a user-defined one. Some computations are automatically done in parallel. For more specific things it will be necessary to write wrapper around existing functions to make use of all the capability of dask.
The catalogs are read making use of the vaex library to handle .fits files. Also, vaex use dask under the hood which make this choice very easy. Also, vaex can open multiple catalogs at once as long as they follow the same format. We will make use of this feature.
Note: by using dask all the computation are "lazy". That means that until you call .compute() method nothing actually happen. Only the tree of the computation is done.
Build a Catalog class to deal with everything related to catalog call. The catalog will be handle through
dask. This will require to write high-level method to make it easier to use and more transparent for the user.Using
daskwill allow to use large catalogs transparently. Also,daskdoes more than catalog handling. The computation on the catalog are done either on aLocalClusteror a user-defined one. Some computations are automatically done in parallel. For more specific things it will be necessary to write wrapper around existing functions to make use of all the capability ofdask.The catalogs are read making use of the
vaexlibrary to handle.fitsfiles. Also,vaexusedaskunder the hood which make this choice very easy. Also,vaexcan open multiple catalogs at once as long as they follow the same format. We will make use of this feature.Note: by using
daskall the computation are "lazy". That means that until you call.compute()method nothing actually happen. Only the tree of the computation is done.vaex.hdf5format to allow memory mapping (not possible from.fitsformat)