de-toolkit: toolkit and sandbox for data engineers

de-toolkit is a collection of open-source software designed to support self-study for data engineers. The kit is built on top of docker and docker-compose. The repository contains docker-compose files that deploy products on the shared de-toolkit-network, along with the configuration files, initialization routines, and examples required to demonstrate how each tool works.

de-toolkit includes tools and systems from categories such as:

• ETL / ELT
• Databases
• DWH management
• BI
• Data Analysis

The project is created and delivered for educational purposes. Don't use de-toolkit in production!

Data Engineer Toolkit

Product	Local ports	Local address	Credentials	Internal hostname
Processing
Airflow	8000	http://localhost:8000	l: de_user, p: de_pass	airflow*
Airbyte	8100, 8101, 8102	http://localhost:8100	l: de_user, p: de_pass	airbyte*
Dagster	8200	http://localhost:8200	no auth	dagster
Prefect2	4200	http://localhost:4200	no auth Docs	prefect*
Spark	8400	http://localhost:8400	no auth — see spark/readme.md	spark, spark://spark:7077
Cronicle	8500	http://localhost:8500	l: admin, p: admin	cronicle
Meltano	5000	http://localhost:5000	no auth	meltano*
Mage	6789	http://localhost:6789	no auth	mage
Storage
PostgreSQL	5432	postgresql://de_user:de_pass@localhost:5432/de	db: de, l: de_user, p: de_pass	postgresql
ClickHouse	8123, 9000	http://localhost:8123/play, clickhouse+http://de_user:de_pass@localhost:8123/de, clickhouse+native://de_user:de_pass@localhost:9000/de	db: de, l: de_user, p: de_pass	clickhouse
MongoDB	27017	mongodb://de_user:de_pass@localhost:27017/de	db: de, l: de_user, p: de_pass	mongodb
Minio	9001,9002	http://localhost:9001	l: de_user, p: de_password	minio
DWH building
dbt	7000	http://localhost:7000	no auth	dbt*
Visualization
Metabase	3000	http://localhost:3000	set user on first start	metabase
Superset	3001	http://localhost:3001	l: de_user, p: de_pass	superset*
Analytics
Jupyter	4000	http://localhost:4000	p: de_pass change password instruction	jupyter
Metadata
Open Metadata	8585,8586, 8587, 9200, 9300	http://localhost:8585	l: admin, p: admin	omd-*
Open Data Discovery	9400	http://localhost:9400	no auth	odd-*

Quick start

git clone https://github.com/iradio/de-toolkit.git
cd de-toolkit

Select a product and run it via docker-compose. Some products must be launched with the build flag, so it is safer to use it by default. For example, to start PostgreSQL:

cd postgresql
docker-compose up -d --build

All products start on the shared de-toolkit-network and are accessible via hostnames that match their service names. For example, when connecting PostgreSQL to Metabase, use the hostname postgresql as the database address.

Requirements

• docker
• docker-compose

Windows - Docker Desktop with WSL 2 backend

Mac - Docker Desktop on Mac

Linux - Docker Desktop on Linux

Ensure Docker has more than 8 GB of memory available.

How to check available docker memory:

docker run --rm "debian:bullseye-slim" bash -c 'numfmt --to iec $(echo $(($(getconf _PHYS_PAGES) * $(getconf PAGE_SIZE))))' 

Tested on Windows 10 Home 64 + Docker Desktop on WSL2. Hardware: Intel i7-10710U 64GB RAM SSD.

Defaults

Environment variables have definitions in the .env file. Each directory has a .env file inside (which can be hidden by default in your OS).
Default credentials:
username: de_user
password: de_pass

Author

• Tim Alein - iradio