# Small EC2 Instance Airflow Tutorial

## EC2 Configuration
- **Instance Type**: T2.small
- **Key Pair**: Create a new key pair
- **Image**: Ubuntu

## Connect to EC2 Instance
After launching your EC2 instance, connect to it in aws


## Commands to Run


   sudo apt update

2. Install Python 3 and pip:

   sudo apt install python3-pip

3. Install SQLite3:

   sudo apt install sqlite3

4. Install the Python 3.10 virtual environment package:

   sudo apt install python3.10-venv

5. Create a Python virtual environment:

   python3 -m venv venv

6. Activate the virtual environment:

   source venv/bin/activate

7. (Optional) Install PostgreSQL development libraries:

   sudo apt-get install libpq-dev

8. Install Apache Airflow with PostgreSQL support:

   pip install "apache-airflow[postgres]==2.5.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.5.0/constraints-3.7.txt"

9. Initialize the Airflow database:

   airflow db init

10. Install PostgreSQL and its contrib package:

    sudo apt-get install postgresql postgresql-contrib

11. Switch to the PostgreSQL user:

    sudo -i -u postgres

12. Access the PostgreSQL shell:

    psql

13. Create the Airflow database and user:

    CREATE DATABASE airflow;
    CREATE USER airflow WITH PASSWORD 'airflow';
    GRANT ALL PRIVILEGES ON DATABASE airflow TO airflow;

14. Exit the PostgreSQL shell by pressing Ctrl + D.

15. Navigate to the Airflow directory:

    cd airflow

16. Replace the connection string in the airflow.cfg file to use PostgreSQL instead of SQLite:

    sed -i 's#sqlite:////home/ubuntu/airflow/airflow.db#postgresql+psycopg2://airflow:airflow@localhost/airflow#g' airflow.cfg

17. Verify the SQL Alchemy connection string:

    grep sql_alchemy airflow.cfg

18. Check the executor configuration:

    grep executor airflow.cfg

19. Replace SequentialExecutor with LocalExecutor:

    sed -i 's#SequentialExecutor#LocalExecutor#g' airflow.cfg

20. Re-initialize the Airflow database:

    airflow db init

21. Create an Airflow user:

    airflow users create -u airflow -f airflow -l airflow -r Admin -e airflow@gmail.com

    • Enter Password: airflow
    • Repeat Password: airflow

22. Update security group inbound rules in your EC2 instance:

    • Type: Custom TCP
    • Port Range: 8080
    • Source: Anywhere (IPV4)
    • Save the rules.

23. Start the Airflow webserver:

    airflow webserver &

24. Start the Airflow scheduler:

    airflow scheduler

25. Copy the public IPv4 DNS of your EC2 instance and open it in your browser with port 8080:

    http://your-ec2-public-ip:8080
    

Use below dag code as a test. make sure to mkdir dag folder and then put this .py script in there!

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime

default_args = {
    'owner': 'airflow',
    'start_date': datetime(2023, 1, 1),
    'retries': 1,
}

dag = DAG(
    'my_new_dag',
    default_args=default_args,
    schedule_interval='@daily',
)

start = DummyOperator(task_id='start', dag=dag)
end = DummyOperator(task_id='end', dag=dag)

start >> end