TODO

[trymzet]

ELT workshops - TODO

0. Intro

• create intro presentation #7

1. Set up

• add VSCode set up info to README #15
  • extensions: Python, Python Docstring Generator, Live Share
• create set up guide #14
• create set up script #13
• test set up on another machine #11
• test VSCode Live with another person #12

2. Part 1 - Prefect Intro

• create notebook code - intro #2
• create notebook instructions - intro #9

3. Part 2 - Build a pipeline - existing Airbyte source

Build a full pipeline, eg. API source -> local filesystem -> Metabase.

• build a diagram with the process #6
  Idea for now: source from Strava using Airbyte, sink to Postgres, use DBT to transform, and set up a simple viz in Metabase.
  See https://github.com/desertaxle/airbyte-prefect-recipe for Prefect + Airbyte + dbt.
  See https://airbyte.io/recipes/build-a-github-activity-dashboard-for-your-project for Airbyte + Metabase.
• add Great Expectations check task #18

4. Part 3 - Build a pipeline - new Airbyte source

Create a new API source and build a full pipeline using it, eg. New API source -> local filesystem -> Metabase using.

• create notebook code - existing source #5
• create notebook instructions - existing source #10

5. General

• choose Superset / Metabase #16
• new source #4
• create general presentation #3
• check if can replace Airbyte with viadot #8
• clean up envs #17

6. Next iterations

• add PR template
• add mkdocs
• add CI/CD
• write a cleanup script
• add prod tips
• VSCode set up script (settings, extensions)
• add part 4 - "Build a library for self-service analytics"
• use custom app to enter data and serve API
• consider Dagster