local setup (MacBook)

Pre-Requisites :

• python interpreter 3.10+ install poetry for dependencies
• Docker account free single user license | Sign-In to Docker Desktop in running state
• Credentials Vault LastPass

$ make env-up

Documentation

• lucidchartArchitectural Overview

cloud platform | hosting

• GCP

scheduler | job orchestration | ELT

• prefect
  • workspace
  • CLI access
  • ref : instructions | setup manual
• mage.ai host details / reach out for ***.pem file

data store | warehouse

• BigQuery
• MongoDB
• PostgreSQL

visualization | consumption

• mode
• preset

data processing

• spark
• DuckDB

transformations | dimension modeling | materialization

• dbt-core

observability | monitoring | data-quality

• metaplane
• soda
• great-expectations