Skip to content

Catalogs handler #3

@aguinot

Description

@aguinot

Build a Catalog class to deal with everything related to catalog call. The catalog will be handle through dask. This will require to write high-level method to make it easier to use and more transparent for the user.
Using dask will allow to use large catalogs transparently. Also, dask does more than catalog handling. The computation on the catalog are done either on a LocalCluster or a user-defined one. Some computations are automatically done in parallel. For more specific things it will be necessary to write wrapper around existing functions to make use of all the capability of dask.
The catalogs are read making use of the vaex library to handle .fits files. Also, vaex use dask under the hood which make this choice very easy. Also, vaex can open multiple catalogs at once as long as they follow the same format. We will make use of this feature.

Note: by using dask all the computation are "lazy". That means that until you call .compute() method nothing actually happen. Only the tree of the computation is done.

  • Read catalog using vaex
    • Allow the convertion to .hdf5 format to allow memory mapping (not possible from .fits format)
    • Store the converted catalog in the workspace directory
    • Make it possible to remove the converted file to save space on disk
    • Keep track of the catalog of origin for each objects
  • Read catalog from config file
  • Make it possible to instantiate a catalog from a path or a list of path
  • Make a parent class
    • Build a children class for galaxy catalog
    • Build a children class for star catalog
  • Handle star catalogs from MCCD (-> convert global positions to local, maybe this should be done in MCCD)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions