MQTT Manager Backend

A comprehensive backend server for managing MQTT servers and monitoring system health. This application provides REST APIs, a web-based dashboard, and automated health monitoring with persistent storage.

Features

• MQTT Server Management: Start, stop, and restart Mosquitto MQTT broker via REST API
• Screen Session Integration: MQTT server runs in detached screen sessions
• System Health Monitoring: Real-time CPU, memory, and disk usage tracking
• Resource Usage Tracking: Monitor MQTT server's CPU and memory consumption
• Persistent Storage: Dual storage support (JSON files + PostgreSQL)
• Automated Status Saving: Status persisted every 15 seconds
• Web Dashboard: Real-time monitoring UI with auto-refresh
• REST API: Complete API with Swagger documentation
• Type Safety: Full type hints throughout the codebase
• Modular Architecture: OOP design with clean separation of concerns
• Comprehensive Tests: Unit tests for all major components

Architecture

backend/
├── src/
│   ├── config/              # Configuration management
│   ├── models/              # Pydantic data models
│   ├── persistence/         # Storage layer (JSON + PostgreSQL)
│   ├── services/            # Business logic (MQTT, health monitoring)
│   ├── routes/              # REST API endpoints
│   ├── static/              # Frontend assets (CSS, JS)
│   └── templates/           # HTML templates
├── tests/                   # Comprehensive test suite
├── data/                    # JSON file storage
├── config.yaml              # Application configuration
└── main.py                  # Application entry point

Requirements

• Python 3.9+
• PostgreSQL (optional, falls back to JSON)
• Mosquitto MQTT broker
• Screen command-line utility

Installation

1. Clone the repository:

   cd backend

2. Create virtual environment:

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

   pip install -r requirements.txt

4. Configure the application: Edit config.yaml to customize:

   • MQTT broker command
   • Server host and port
   • Database credentials
   • Monitoring intervals

Configuration

The config.yaml file controls all aspects of the application:

mqtt:
  command: "mosquitto -c /etc/mosquitto/mosquitto.conf"
  screen_session_name: "mqtt_server"
  port: 1883
  host: "localhost"

server:
  host: "0.0.0.0"
  port: 8080
  debug: true

persistence:
  json_file: "data/mqtt_status.json"
  postgres:
    host: "localhost"
    port: 5432
    database: "mqtt_manager"
    user: "postgres"
    password: "postgres"

monitoring:
  status_save_interval: 15  # seconds

system:
  cpu_threshold: 80
  memory_threshold: 80

Running the Application

Option 1: Docker (Recommended)

The easiest way to run the application is using Docker Compose:

1. Build and start all services:

   docker-compose up -d

2. View logs:

   docker-compose logs -f mqtt_manager

3. Stop services:

   docker-compose down

4. Using Makefile shortcuts:

   make up       # Start all services
   make logs     # View logs
   make down     # Stop all services
   make restart  # Restart services
   make clean    # Remove containers and volumes
   make test     # Run tests in container
   make shell    # Open shell in container

Option 2: Local Development

1. Start the server:

   python main.py

2. Access the dashboard:

   • Dashboard: http://localhost:8080/
   • API Documentation: http://localhost:8080/api/docs
   • Health Check: http://localhost:8080/health

Docker Services

The Docker Compose setup includes:

• mqtt_manager: Flask application (port 8080)
• postgres: PostgreSQL database (port 5432)
• Persistent volumes for database data
• Health checks for all services
• Automatic restart on failure

API Endpoints

MQTT Management

• POST /api/mqtt/start - Start MQTT server
• POST /api/mqtt/stop - Stop MQTT server
• POST /api/mqtt/restart - Restart MQTT server
• GET /api/mqtt/status - Get current MQTT status

System Health

• GET /api/health/current - Get current system health
• GET /api/health/mqtt - Get latest MQTT health from storage
• GET /api/health/history/system?limit=100 - Get system health history
• GET /api/health/history/mqtt?limit=100 - Get MQTT status history
• GET /api/health/combined - Get combined health data

Testing

Run the test suite:

# Run all tests
pytest

# Run with coverage
pytest --cov=src --cov-report=html

# Run specific test file
pytest tests/test_config.py

Project Structure Details

Configuration Layer (src/config/)

• ConfigManager: Centralized configuration with dot-notation access
• Supports YAML configuration files
• Hot-reload capability

Data Models (src/models/)

• MQTTStatus: Type-safe MQTT server status model
• SystemHealth: System resource usage model
• Built with Pydantic for validation

Persistence Layer (src/persistence/)

• PersistenceStore: Abstract base class
• JSONStore: File-based storage implementation
• PostgresStore: PostgreSQL database implementation
• Automatic fallback from PostgreSQL to JSON

Services (src/services/)

• MQTTManager: MQTT server lifecycle management
• HealthMonitor: System resource monitoring
• StatusSaver: Periodic status persistence (background thread)

Routes (src/routes/)

• mqtt_routes: MQTT server control endpoints
• health_routes: Health monitoring endpoints
• Factory pattern for blueprint creation

Frontend

• Plain HTML/CSS/JavaScript
• Auto-refreshing dashboard (5-second interval)
• Real-time progress bars and status indicators
• Responsive design

Design Patterns

1. Dependency Injection: Services receive dependencies via constructors
2. Factory Pattern: Route blueprints created via factory functions
3. Strategy Pattern: Persistence layer with interchangeable backends
4. Observer Pattern: Background thread for periodic status updates
5. Singleton Pattern: Configuration manager instance

Best Practices

• Type hints throughout the codebase
• Comprehensive docstrings (Google style)
• Clean code principles (DRY, SOLID)
• Error handling and logging
• Resource cleanup on shutdown
• Thread-safe operations

Troubleshooting

Docker Issues

Container won't start:

# Check logs
docker-compose logs mqtt_manager

# Rebuild from scratch
docker-compose down -v
docker-compose build --no-cache
docker-compose up -d

Database connection issues:

# Check PostgreSQL is running
docker-compose ps postgres

# Check database logs
docker-compose logs postgres

# Restart database
docker-compose restart postgres

Port conflicts: If ports 8080 or 5432 are already in use, modify docker-compose.yml:

ports:
  - "8081:8080"  # Change external port

PostgreSQL Connection Failed

The application automatically falls back to JSON storage if PostgreSQL is unavailable.

MQTT Server Won't Start

• Check that Mosquitto is installed: mosquitto -h
• Verify the command in config.yaml
• Check screen is installed: screen --version
• Review logs for specific errors

Dashboard Not Updating

• Check browser console for API errors
• Verify backend is running
• Check CORS settings

License

See LICENSE file in the repository root.

Contributing

This project follows clean code principles. When contributing:

• Add type hints to all functions
• Write docstrings for public methods
• Add tests for new features
• Follow existing code style
• Update documentation as needed