Selimetry 🚀

A High-Performance, Scalable Observability & Monitoring Platform

Selimetry is a modern, centralized logging and metrics aggregation system designed to provide real-time insights into your application's health and performance. Built with a microservices architecture, it seamlessly ingests logs, calculates metrics on the fly, and visualizes data for actionable intelligence.

---

✨ Key Features

• ⚡ High-Throughput Ingestion: A Go-based ingestor service capable of handling high volumes of log data via REST API.
• 📊 Automated Metrics Aggregation: Background services automatically calculate key metrics like Request Count, Error Rate, and Response Times.
• 📉 Smart Storage: Optimizes database usage by only storing metrics when significant changes occur.
• 🔍 Flexible Querying: Powerful API to filter logs by service, level, environment, and time range.
• 📈 Visualization Ready: Pre-configured integration with Grafana for beautiful, real-time dashboards.
• 🐳 Dockerized: Fully containerized setup for easy deployment and scaling.

---

�🚀 Performance Benchmarks

The Go-based Ingestor is optimized for high throughput. Below are the benchmark results for processing 50,000 requests under different batching configurations:

Batch Size	Total Time	Avg Latency	Requests/sec
No Batching	3.2056s	3.2ms	15,597
50	4.3019s	4.2ms	11,622
1,000	2.4674s	2.4ms	20,264
5,000	2.4065s	2.4ms	20,777

Note: Tests were conducted on a local environment. Batching significantly improves throughput, peaking at over 20,000 requests per second with larger batch sizes.

---

🏗️ System Architecture

---

🛠️ Technology Stack

• Ingestion Layer: Go (Golang) - Chosen for its raw performance and concurrency.
• Processing Layer: ASP.NET Core 8.0 - Robust, type-safe backend for complex aggregation logic.
• Message Broker: Apache Kafka - Ensures reliable, asynchronous data streaming.
• Database: PostgreSQL 16 - Reliable relational storage for logs and time-series metric data.
• Visualization: Grafana - Industry-standard tool for monitoring dashboards.
• Infrastructure: Docker & Docker Compose.

---

🚀 Getting Started

Prerequisites

• Docker
• Docker Compose

Installation & Running

1. Clone the repository:

   git clone https://github.com/abdallahsellem/Selimetry.git
   cd Selimetry

2. Start the services:

   docker-compose up -d

3. Access the services:

   • Backend API (Swagger): http://localhost:8080/swagger
   • Ingestor API: http://localhost:8081
   • Grafana: http://localhost:3000 (Default login: admin / admin)

---

📦 Services Overview

1. SelimetryIngestor (Go)

• Port: 8081
• Role: The entry point for all log data. It validates incoming requests and rapidly produces them to a Kafka topic.
• Key Tech: Go, Sarama (Kafka Client).

2. SelimetryConsumer (.NET)

• Port: 8080
• Role: Consumes messages from Kafka, persists them to the database, and runs background jobs to aggregate data into useful metrics.
  • Key Tech: .NET 8, Entity Framework Core, Hosted Services.

---

📂 Project Structure

Selimetry/
├── SelimetryServices/
│   ├── SelimetryConsumer/    # .NET 8 Backend Service (Consumer & Aggregator)
│   └── SelimetryIngestor/    # Go High-Throughput Ingestor Service
├── infrastructure/           # Infrastructure configurations (Terraform - Coming Soon)
├── docker-compose.yml        # Container orchestration
└── README.md                 # Project documentation

---

⚙️ Configuration

The entire platform is configured via docker-compose.yml. Key environment variables include:

• Database Credentials: Configured in the db service and consumed by backend.
• Kafka Brokers: Configured in kafka and used by both ingestor and backend.

To modify database passwords or ports, update the .env file (if present) or directly edit docker-compose.yml.

---

Generated Metrics

Selimetry automatically aggregates logs to calculate the following real-time metrics for each service:

Metric Name	Description
Request Count	Total volume of requests processed by the service.
Error Rate	The percentage of logs marked as Error or Critical.
Exception Rate	The frequency of unhandled exceptions logged by the service.
Logs Per Minute	A throughput metric indicating the rate of incoming log entries.
Activity By Hour	Analyzes usage patterns to show activity distribution throughout the day.
Health Score	A composite score (0-100) indicating the overall health of the service based on errors and latency.

---

📝 API Documentation

Once the services are running, you can explore the full API specification for the Consumer service via Swagger UI:

👉 Open Swagger UI

Example: Ingest a Log

curl -X POST http://localhost:8081/ingest \
  -H "Content-Type: application/json" \
  -d '[{
    "serviceName": "PaymentService",
    "level": "Error",
    "message": "Transaction failed",
    "timestamp": "2024-12-10T10:00:00Z",
    "environment": "Production"
  }]'

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

---

🗺️ Roadmap & Future Improvements

• Infrastructure as Code: Implement Terraform scripts for provisioning cloud resources (AWS/Azure) in the infrastructure/ folder.

• CI/CD Pipeline: Set up GitHub Actions for automated testing and deployment.

• Enhanced Visualization: Add more granular Grafana dashboards for specific service metrics.

• Alerting: Configure Prometheus Alertmanager for critical system events.

📄 License

This project is proprietary software.