diff --git a/docs/DEVOPS.md b/docs/DEVOPS.md new file mode 100644 index 0000000..91a46c9 --- /dev/null +++ b/docs/DEVOPS.md @@ -0,0 +1,251 @@ +# Project Delta: DevOps & Infrastructure Engineering + +A comprehensive technical overview of the DevOps lifecycle, architectural components, and automation pipelines implemented in Project Delta. + +--- + +## 1. System Architecture & Orchestration + +Project Delta is built on a monolithic-backend architecture, fully containerized using **Docker**. This ensures a consistent runtime environment across all development and production stages. + +### Infrastructure Overview (Docker Compose) + +The entire backend stack is defined and managed via `docker-compose.yml`. This configuration handles service isolation, networking, and volume persistence. + +#### `docker-compose.yml` Configuration: +```yaml +services: + postgres: + image: postgres:18-alpine + container_name: delta-postgres + restart: on-failure + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: 1234 + POSTGRES_DB: postgres + ports: + - "5432:5432" + volumes: + - postgres_data:/var/lib/postgresql/data + + redis: + image: redis:7-alpine + container_name: delta-redis + restart: on-failure + ports: + - "6379:6379" + volumes: + - redis_data:/data + command: redis-server --appendonly yes + + drizzle-gateway: + image: ghcr.io/drizzle-team/gateway:latest + container_name: delta-drizzle-gateway + environment: + DATABASE_URL: postgresql://postgres:1234@postgres:5432/postgres + STORE_PATH: /app + restart: on-failure + depends_on: + - postgres + ports: + - "4983:4983" + volumes: + - drizzle_gateway_data:/app + + api: + build: . + container_name: delta-api + restart: always + ports: + - "8000:8000" + depends_on: + - postgres + - redis + env_file: + - .env + volumes: + - repos_data:/repos + + worker: + build: . + container_name: delta-worker + restart: always + command: python workers.py + depends_on: + - redis + - postgres + env_file: + - .env + volumes: + - repos_data:/repos + +volumes: + postgres_data: + redis_data: + drizzle_gateway_data: + repos_data: +``` + +### Service Breakdown: +1. **API (FastAPI)**: The primary application server handling HTTP traffic. It relies on `postgres` for data and `redis` for caching. +2. **RQ Worker**: A Celery/standalone background processor that executes long-running tasks asynchronously. +3. **PostgreSQL**: A relational database for persistent storage, using the `18-alpine` image for a minimal footprint. +4. **Redis**: Handles message brokering and session management with `appendonly` persistence enabled. + +--- + +## 2. CI/CD Lifecycle (GitHub Actions) + +CI/CD pipeline ensures that every code change is validated before reaching production. This automated feedback loop is critical in a collaborative environment to prevent regressions and maintain high code quality standards. + +### Continuous Integration (`backend-ci.yml`) + +The CI pipeline runs on every `push` and `pull_request` to the `main` branch. Its goal is to catch bugs, security vulnerabilities, and formatting issues early in the development cycle. By automating these checks, we ensure that the `main` branch always remains in a deployable state. + +```yaml +name: Backend CI + +on: + push: + branches: [ "main" ] + pull_request: + branches: [ "main" ] + +jobs: + ruff-lint: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up Python 3.12 + uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install ruff + run: pip install ruff + - name: Run ruff check + run: ruff check + + pyrefly-typecheck: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Set up Python 3.12 + uses: actions/setup-python@v5 + with: + python-version: "3.12" + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + - name: Install pyrefly + run: pip install pyrefly + - name: Run pyrefly + run: pyrefly check + + backend-tests: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v4 + - name: Set up Python 3.12 + uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: 'pip' + + - name: Install dependencies + run: | + python -m pip install --upgrade pip + pip install -r requirements.txt + + - name: Set up environment + run: cp .env.example .env + + - name: Run tests + run: python -m pytest +``` + +--- + +### GitHub Repository Secrets + +For security and privacy, sensitive data such as server IP addresses, login credentials, and private SSH keys are neve stored in the codebase. Instead, they are managed via **GitHub Repository Secrets**. + +These secrets are encrypted environment variables that GitHub Actions can access during runtime but are hidden from all users, including repository contributors. + +**Key Secrets Configured:** +- `VPS_IP`: The public IP address of the production server. +- `VPS_USERNAME`: The administrative user authorized to perform deployments. +- `VPS_SSH_KEY`: The private RSA/ED25519 key used for secure, passwordless SSH authentication. + +--- + +### Continuous Deployment (`deploy.yml`) + +Once the CI pipeline passes, the CD pipeline automates the rollout to our production VPS. + +```yaml +name: Deploy to VPS +on: + push: + branches: [ "main" ] +jobs: + deploy: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Deploy via SSH + uses: appleboy/ssh-action@v1.0.3 + with: + host: ${{ secrets.VPS_IP }} + username: ${{ secrets.VPS_USERNAME }} + key: ${{ secrets.VPS_SSH_KEY }} + script: | + # 1. Enter the project folder + cd delta.backend + + # 2. Pull the latest code + git pull origin main + + # 3. Build and run updated containers + sudo docker-compose up --build -d + + # 4. Migrate the DB + sudo docker-compose exec -T api alembic upgrade head +``` + +#### Deployment Mechanisms: +- **Git Hook**: Pulls the latest delta from the repository. +- **Docker-Compose Logic**: The `--build -d` flags ensure that any changes to the `Dockerfile` or source code are rebuilt into new images and restarted without downtime. + +--- + +## 3. Operational Infrastructure + +### Frontend (Vercel) +The React/TypeScript application is hosted on **Vercel**. +- **Edge Deployment**: Static assets are distributed globally. +- **Vercel Rewrites**: Configured in `vercel.json` to route all traffic to `index.html`, enabling client-side navigation. + +### Backend (VPS-Amazon EC2) +Hosted on a dedicated **Ubuntu/Linux VPS**. +- **Orchestration**: Docker Compose manages container life-cycles. +- **Security**: SSH Key authentication is required for all deployment triggers. + +--- + +## 4. Engineering Maintenance + +### Configuration Management +All sensitive credentials must be stored in a `.env` file. Do NOT commit the `.env` file to version control. +```bash +# Apply migrations after a schema change +docker-compose exec api alembic upgrade head +``` + +### Health & Monitoring +```bash +# View live logs for specific service +docker-compose logs -f api +```