🌤️ Climate Data Service

Service for collecting climate data, implementing an ETL pipeline to monitor weather conditions in different Brazilian cities.

🏗️ Architecture

The project uses Docker containers to orchestrate:

• ETL Service: Python pipeline for weather data collection
• REST API: Spring Boot application to expose the data
• PostgreSQL: Database for storage

🚀 How to Run

Prerequisites

• Docker and Docker Compose installed
• WeatherAPI.com API key (free)

1. Initial Setup

# Clone the repository
git clone <your-repository>
cd climate-data-service

# Copy the environment variables example file
cp env.example .env

# Edit the .env file and add your API key
nano .env

2. Configure Environment Variables

Edit the .env file and add your API key:

API_KEY=your_api_key_here

Get your free key at: https://www.weatherapi.com/

3. Run with Docker Compose

# Build and start all services
docker-compose up -d

# View ETL service logs
docker-compose logs -f climate-etl

# View PostgreSQL logs
docker-compose logs -f postgres

4. Access the Services

• REST API: http://localhost:8081
  • Swagger UI: http://localhost:8081/swagger-ui.html
  • Health Check: http://localhost:8081/health
• PostgreSQL: localhost:5432

📊 Monitoring

Check Container Status

docker-compose ps

View Real-Time Logs

# All services
docker-compose logs -f

# Only ETL
docker-compose logs -f climate-etl

Check Data in Database

# Connect to PostgreSQL
docker-compose exec postgres psql -U climate_user -d climate_db

# Query data
SELECT * FROM weather ORDER BY created_at DESC LIMIT 10;

# Run useful queries
\i data/queries.sql

Test the REST API

# List all records
curl http://localhost:8081/clima

# Search by city
curl http://localhost:8081/clima/Campinas

# Get today's data for a city
curl http://localhost:8081/clima/Campinas/hoje

# API health check
curl http://localhost:8081/health

🛠️ Useful Commands

# Stop all services
docker-compose down

# Stop and remove volumes (warning: deletes data)
docker-compose down -v

# Rebuild containers
docker-compose up --build -d

# Run only the ETL (without scheduling)
docker-compose run --rm climate-etl python etl_project/etl_pipeline.py

# Backup the database
docker-compose exec postgres pg_dump -U climate_user -d climate_db > data/backup_$(date +%Y%m%d_%H%M%S).sql

📁 Project Structure

climate-data-service/
├── etl_project/          # ETL pipeline code
│   ├── config/          # Configurations
│   ├── utils/           # Utilities
│   ├── main.py          # Entry point
│   ├── etl_pipeline.py  # Main pipeline
│   ├── Dockerfile       # ETL app container
│   └── requirements.txt # Python dependencies
├── api/                  # Spring Boot application
│   ├── src/main/java/   # Java code
│   ├── src/main/resources/ # Configurations
│   ├── src/test/        # Tests
│   ├── Dockerfile       # API container
│   └── pom.xml          # Maven dependencies
├── data/                # Database files
│   ├── init.sql         # Initialization script
│   ├── backup.sql       # Backup script
│   ├── queries.sql      # Useful queries
│   └── README.md        # Data documentation
├── logs/                # Application logs
├── docker-compose.yml   # Container orchestration
├── env.example         # Environment variable example
└── .gitignore          # Git ignored files

🔧 Configuration

Monitored Cities

Edit etl_project/config/config.yaml to change the cities:

cities:
  - Campinas
  - Limeira
  - Americana
  - Sao Paulo
  - Rio de Janeiro

Collection Frequency

In etl_project/config/config.yaml, change the interval:

schedule:
  interval:
    value: 30
    unit: seconds

📈 Collected Data

The system automatically collects:

• Temperature (Celsius)
• Wind speed (km/h)
• Relative humidity (%)
• Location (city, region, country)
• Collection timestamp

🐛 Troubleshooting

Problem: ETL container does not start

# Check logs
docker-compose logs climate-etl

# Check if PostgreSQL is healthy
docker-compose ps postgres

Problem: Database connection error

# Check if environment variables are correct
docker-compose exec climate-etl env | grep POSTGRESQL

Problem: API does not respond

• Check if the API key is correct in the .env file
• Confirm the API is working: https://www.weatherapi.com/

📝 Logs

Logs are saved in:

• ./logs/ (mounted volume in the container)
• Container console (via docker-compose logs)

🔒 Security

• Database credentials are set in docker-compose.yml
• The API key must be kept safe in the .env file
• The .env file should not be committed to the repository

📞 Support

For issues or questions:

1. Check the logs: docker-compose logs
2. See WeatherAPI.com documentation
3. Open an issue in the repository

🎯 Challenge Compliance

✅ Requirements Met:

1. ETL with Python ✅

   • Python 3 script consuming weather API
   • Configurable city query
   • Data treatment and validation
   • Periodic scheduling

2. PostgreSQL ✅

   • Relational database structure
   • Creation scripts (init.sql)
   • Integrity and normalization

3. REST API with Spring Boot ✅

   • GET /clima - List all records
   • GET /clima/{cidade} - Filter by city
   • GET /clima/{cidade}/hoje - Most recent data of the day

4. Containerization ✅

   • Python container (ETL)
   • PostgreSQL container
   • Spring Boot container (API)
   • docker-compose.yml for orchestration

🚀 Implemented Differentials:

• Swagger Documentation: Interactive API interface
• Health Checks: Service health monitoring
• Structured logs: Easier debugging
• Automated tests: Test structure configured
• Multi-stage Docker builds: Image optimization

---

🇧🇷 Leia a documentação em português

🤝 Contributing

See the contribution guidelines.

📝 License

This project is licensed under the MIT License.