Data Extraction and Load is my first beginner project in the are of data engineering. This project is a part of the Borderless Incubator project on forecasting cryptocurrency price. The whole project consist of four teams and I lead the data engineering team.
The data engineering team was tasked to scrape cryptocurrency data from different live source and store in a database. Our work flow is to extract, clean and load the data into a cloud base database system.
As this project for learning we at first set up a database locally and then we migrated to the cloud.
We were also task to automate our exctraction script to run every one hour.
Our choice of tools and service in the data engineering team are listed asunder. As this is still and on going project we will continue to update the work we've done
- Cloud services: Azure Database for Postgres; Azure Data Studio,
- Local DB: PostgreSql Server
- Webscraping: requests; Beautiful Soup
- SQL queries: sqlalchemy; pandas
- Language: Python; SQL
- Environment: Jupyter Notebook; Spyder; pgAdmin
- Other Modules and Packages: time module; datetime module; csv; os