Intermediate DAG representation layer

[Kitchi]

Create functionality that can hold an abstract representation of a DAG, using Python graphlib or similar.

Briefly : The idea is to be able to express any arbitrary DAG as a dict (or similar) such that we can then use this generic representation to get more specific, and write out the appropriate SLURM or HTCondor representation. The advantage of such a structure is that in principle we should be able to use this intermediate layer to ingest a DAG representation that is output from a different system (such as Dask for e.g.,).

This should in principle allow a user to express a DAG in one system, and deploy that same workflow on a cluster backend such as HTCondor without too many changes.

Dask has a HighLevelGraph implementation that holds the internal representation of a Dask DAG. We can either adopt the same convention, or use it as a starting point and create something more specific to radio astronomy.

In the HighLevelGraph structure, they express both nodes and edges in the dictionary.
In the RA context, a "node" of a DAG is a particular function/algorithm, and an "edge" of a DAG is the data that flows in and out. Therefore we must be able to use the edge definition to capture our data products that go into and out of a given script/function/algorithmic step.

References :

• Python graphlib
• Dask HighLevelGraph