Large Refactor

[Kitchi]

This issue tracks a large scale refactor that is necessary to the codebase, after more thought about the structure of the code and the required outcomes.

The overview of the plan for the new layout is below :

# Layout of classes

- Keep the classes general, specializing as late as possible.
- We want a generic DAG Graph layer to hold the JSON representation of the DAG,
  and specialize down to SLURM and Condor downstream. 
- Keep a track of which data needs to be transferred to the EP (for condor, can
  ignore for SLURM)
- Ability to specify a software runtime, and ancillary files that need to be
  transferred. The ancillary files are not necessarily inputs to the functions.
- Translate a Python function into a standalone script, potentially with
  argparse with the arguments in the same order as the input args.
- Use decorators to wrap the Python functions into the generic DAG.

# Classes

- Layer       : dataclass; Holds information about a single DAG layer

- DAGGraph    : Class to build a generic DAG, something that is compatible with
                Python graphlib. Need to be able to : 
                + Add layer (from Layer) and all the info that needs such as the input function, arguments, exec environment.
                  Similar to the existing Dagger.add_function_to_layer.
                + Specify layer dependencies in both directions (parents and children)
                + Go to and from a Python dict to reload and save a DAG
                + Convenience functions to write out the target functions to Python scripts, and convert arguments that will be passed in to command line args 
                  via argparse.


- CondorDAG   : Inherits from DAGGraph, and uses the HTCondor Python library to interface with the DAGMAN functionality of HTCondor, writing out Python scripts and 
                corresponding DAGMAN files.
                + Needs to have a way to specify condor submit parameters, such as resource usage, singularity paths, and which files to transfer in and out.
                + Pass in the arguments from the target function via the command line within the condor submit scripts.
                + Write out appropriate submit scripts, passing in relevant arguments via command line. Make sure if we expect a file to exist at the EP, we have it in
                  input_transfer_files


- SlurmDAG    : Inherits from DAGGraph, and writes out SLURM submit files and associated Python scripts. 
                + Similar to CondorDAG, specify submit parameters such as resource usage, singularity execution environments.
                + For SLURM we assume a shared filesystem, so if input/output transfer files are set, silently ignore them after emitting a warning to the user.
                + Need functionality to write SLURM submit files.


- WorkflowDAG : A high level wrapper around CondorDAG and SlurmDAG. Allows user to define a backend-independent DAG, like WorkflowDAG(backend='slurm')
                + Transparent access to all the layer adding, submit param specifying functions. 
                + Have a single write function that writes out the relevant submit files based on the backend selected.

This issue will close several other issues upon completion. namely #10, #9, #8, #6, #2.