This project relies on docker and terraform. Please run the commands below in the order encountered

To set up terraform run

cd terraform (navigate to the terraform directory)

terraform init

To initialise the airflow db

docker compose up airflow-init

Now all services can be started

docker-compose up -d

The user must go into localhost(http://localhost:8080/home) at this stage and add a connection, add the following (username and password are both airflow):

Conn Id: local_db Conn type: Postgres Host: host.docker.internal Port: 5433

The required input data should now be placed in terraform/data

To run the DAG execute

docker compose run airflow-worker dags test monolith

To run a particular task run

docker compose run airflow-worker tasks test monolith my_first_dag 2022-02-02

To terminate services at any point run

docker compose down