Customer Data ETL Pipeline (SQL Server + Docker)

This project simulates a real-world ETL pipeline for cleaning, merging, and auditing customer data from multiple sources.

🔧 Tech Stack

• Microsoft SQL Server (Docker)
• T-SQL (sqlcmd)
• Bulk insert from CSV
• Stored Procedures for automation

📁 Project Structure

• data/: Sample input data (customer_data.csv)
• sql/: SQL scripts for table creation, ETL logic, and testing
• screenshots/: Optional screenshots or dashboard previews
• README.md: Project overview

✅ ETL Process

1. Load raw customer data into Staging_Customers
2. Clean inconsistent city names (e.g. new york → New York)
3. Merge data into Dim_Customers using UPSERT logic
4. Log each ETL run in ETL_Job_Log
5. Clear staging table

🧪 Sample Queries

• Active vs inactive customer count
• Top cities by customer count
• ETL run history

🚀 To Run the Project

# Connect to the DB
sqlcmd -S localhost -U SA -P 'YourStrongPassword' -d Company_Datawarehouse

# Create tables
:i sql/create_tables.sql

# Load your CSV via BULK INSERT or bcp

# Run ETL
EXEC Run_Customer_ETL;
GO

# Run test queries
:i sql/test_queries.sql