Initially created for a Apache Airflow 2.3, in the process of building an Airflow 3.1 training.
Very useful if you are still using the older version, especially the tuning.sh has parameters that need to be changed in GKE or VM based setups.
All new work would be put in here.
Cloud composer is the managed version of Apcahce Airflow and easiest way to get started. But it is not the ideal solution from a cost perspective, so we will install it on a single VM instance on Google Cloud.
We will extract the data from IBDB public website everyday, convert the TSV files into parquet and load into BQ using Apache beam using python as well as Java.
All working except
- bq_ext.tf
- Issue with authentication on macbook with json key.