Skip to content

infrastucture-as-code.md

Latest commit

 

History

History
78 lines (61 loc) · 2.88 KB

infrastucture-as-code.md

File metadata and controls

78 lines (61 loc) · 2.88 KB

Infrastructure as Code (IAC)

Sometimes clicking through the UI is challenging and prone to mistakes. Use Terraform and custom modules to apply the same changes!

If you're starting fresh, check out fresh-start before beginning with this exercise.

  1. Setup
  2. Data Ingestion IAC
  3. Data Transformation IAC
  4. Data Workflow IAC
  5. Destroy everything

Setup

  1. Install dependencies
  2. Ensure that you have an existing S3 bucket, data sources, and artifacts (main.py, .egg). If not check out fresh-start.
  3. Delete all manually created resources from the data-ingestion and data-transformation sections if you created them in a previous exercise (e.g. Glue Job, Crawler, IAM roles)
  4. Ensure you have an active AWS CLI Session

Install Depenencies

Non-Containerised

  1. Install Python (version 3.7), for example with Pyenv
  2. Install Terraform (required version 1.1.3)

Containerised

docker run -it -v $(pwd):/app ghcr.io/data-derp/python-aws-terraform-container:master bash

All workloads can be run within this container once you have set up an active AWS CLI Session

Data Ingestion IAC

cd iac/data-ingestion

# Change these variables
export PROJECT_NAME=awesome-project
export MODULE_NAME=awesome-module
export AWS_DEFAULT_REGION=eu-central-1 (or another valid AWS region)

terraform init && terraform apply -var "project-name=${PROJECT_NAME}" -var "module-name=${MODULE_NAME}" -auto-approve

Data Transformation IAC

cd iac/data-transformation

# Change these variables
export PROJECT_NAME=awesome-project
export MODULE_NAME=awesome-module
export AWS_DEFAULT_REGION=eu-central-1

terraform init && terraform apply -var "project-name=${PROJECT_NAME}" -var "module-name=${MODULE_NAME}" -auto-approve

Data Workflow IAC

cd iac/data-workflow

# Change these variables
export PROJECT_NAME=awesome-project
export MODULE_NAME=awesome-module
export AWS_DEFAULT_REGION=eu-central-1

terraform init && terraform apply -var "project-name=${PROJECT_NAME}" -var "module-name=${MODULE_NAME}" -auto-approve

Run the Workflow in the AWS Console and watch it turn green!

Destroy Everything

To destroy all of the resources you created using Terraform, run this from the root of the repository:

export AWS_DEFAULT_REGION=eu-central-1

for p in data-workflow data-transformation data-ingestion
do
 pushd "./iac/${p}" > /dev/null
    terraform destroy -var "project-name=${PROJECT_NAME}" -var "module-name=${MODULE_NAME}" -auto-approve
 popd > /dev/null
done