Beacon: Real-time Log Aggregation System

Beacon is a high-performance, real-time log aggregation system built with Elixir, designed to compete with ELK stack while offering better real-time capabilities and resource efficiency. It can handle millions of log entries per second with true real-time processing and alerting.

✨ Features

• High-Performance Ingestion: Ingest millions of logs per second
• Real-Time Processing: Process and filter logs in real-time
• Advanced Search: Full-text search with filtering capabilities
• WebSocket Support: Stream logs directly to clients in real-time
• Time Series Storage: Efficient storage with built-in retention policies
• Subscription Management: Create and manage filtered log subscriptions
• Intelligent Alerts: Configurable alert rules with multiple notification channels
• Distributed Architecture: Multi-node clustering with automatic load balancing
• Production Monitoring: Built-in Prometheus metrics, Grafana dashboards, and health checks
• Resource Efficient: Built on Elixir/OTP for excellent performance

📋 Table of Contents

• Architecture
• Getting Started
  • Prerequisites
  • Installation
  • Running the Application
• Usage
  • HTTP API
  • WebSocket API
  • Frontend Dashboard
• Monitoring & Operations
  • Health Checks
  • Metrics & Prometheus
  • Grafana Dashboards
  • Configuration Management
• Distributed Architecture
• Advanced Features
  • Search Capabilities
  • Real-time Streaming
  • Alert System
• Development
  • Project Structure
  • Testing
• Documentation
• Contributing
• License

🏛️ Architecture

Beacon is built on a GenStage processing pipeline architecture with distributed clustering support:

1. Ingestor: Receives and validates incoming logs
2. Parser: Parses and normalizes log formats
3. Pipeline: Multi-stage processing pipeline
   • Filter: Removes unwanted logs
   • Enrichment: Adds metadata and context
   • Aggregation: Performs real-time aggregations
4. Storage: Efficient time-series storage with text indexing and replication
5. Alerts Engine: Monitors logs against configurable rules and triggers notifications
6. Distributed Layer: Node clustering, load balancing, and data replication
7. HTTP/WebSocket API: Unified interface for queries, real-time streaming, and alert management
8. Monitoring: Built-in metrics, health checks, and observability

🚀 Getting Started

Prerequisites

Backend Requirements

• Erlang/OTP 26 or later
• Elixir 1.18 or later

Frontend Requirements

• Node.js 20 or later
• npm 10 or later

Optional (for monitoring)

• Docker & Docker Compose (for Prometheus/Grafana stack)

Installation

Clone the repository:

git clone https://github.com/Chahine-tech/beacon.git
cd beacon

Backend Setup

cd api
mix deps.get

Frontend Setup

cd front
npm install

Running the Application

Start the Backend Server

cd api
mix run --no-halt
# Or for interactive mode:
# iex -S mix

The server will start on port 4000 by default.

Start the Frontend Development Server

cd front
npm run dev

📝 Usage

HTTP API

Ingesting Logs

# Single log with full metadata
curl -X POST http://localhost:4000/logs \
  -H "Content-Type: application/json" \
  -d '{
    "level": "error",
    "message": "Database connection timeout",
    "source": "db-service",
    "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
    "user_id": "12345",
    "request_id": "abc-123"
  }'

# Batch logs
curl -X POST http://localhost:4000/logs/batch \
  -H "Content-Type: application/json" \
  -d '[
    {
      "level": "error",
      "message": "Failed authentication attempt",
      "source": "auth-service",
      "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"
    },
    {
      "level": "info",
      "message": "User profile updated",
      "source": "user-service",
      "timestamp": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
      "user_id": "67890"
    }
  ]'

Searching Logs

# Basic search
curl "http://localhost:4000/search?q=error"

# Search with time range (ISO8601 format)
curl "http://localhost:4000/search?q=database&start_time=2025-01-01T00:00:00.000Z&end_time=2025-01-02T00:00:00.000Z"

# Advanced search with multiple parameters
curl "http://localhost:4000/search?q=error&limit=50&sort=timestamp"

Managing Alerts

# Get alert engine statistics
curl http://localhost:4000/alerts/stats

# Create an alert rule
curl -X POST http://localhost:4000/alerts/rules \
  -H "Content-Type: application/json" \
  -d '{
    "name": "High Error Rate Alert",
    "description": "Triggers when error logs are detected",
    "severity": "high",
    "status": "active",
    "conditions": [
      {
        "type": "threshold",
        "field": "level",
        "operator": "eq",
        "value": "error"
      }
    ],
    "logical_operator": "and",
    "cooldown_seconds": 60,
    "tags": ["production", "critical"]
  }'

# List all alert rules
curl http://localhost:4000/alerts/rules

# Get specific alert rule
curl http://localhost:4000/alerts/rules/{rule_id}

# List alert instances
curl http://localhost:4000/alerts/instances

# Acknowledge an alert instance
curl -X POST http://localhost:4000/alerts/instances/{instance_id}/acknowledge \
  -H "Content-Type: application/json" \
  -d '{
    "acknowledged_by": "operator@example.com",
    "reason": "Investigating the issue"
  }'

WebSocket API

Connect to the WebSocket endpoint to receive logs in real-time:

# Using wscat (npm install -g wscat)
wscat -c ws://localhost:4000/ws/logs

Frontend Dashboard

Access the frontend dashboard in your browser:

http://localhost:5173/

The dashboard provides:

• Real-time log streaming view
• Advanced search interface
• Log analytics and visualization
• System health monitoring

Monitoring & Operations

Beacon includes enterprise-grade monitoring and operations features for production deployments.

Health Checks

Multiple health check endpoints for different use cases:

# Quick liveness check (for load balancers)
curl http://localhost:4000/health

# Readiness check (returns 503 if not ready)
curl http://localhost:4000/health/ready

# Detailed health status (comprehensive diagnostics)
curl http://localhost:4000/health/detailed

Metrics & Prometheus

Export metrics in Prometheus format for monitoring and alerting:

# Prometheus text format
curl http://localhost:4000/metrics

# JSON format
curl http://localhost:4000/metrics/json

Key Metrics:

• logs_ingested_total - Total logs ingested
• logs_ingestion_rate - Logs per second
• storage_bytes_total - Storage size
• system_memory_total_bytes - Memory usage
• cluster_nodes_total - Number of cluster nodes
• realtime_subscriptions - Active WebSocket connections
• alerts_active_rules - Active alert rules
• alerts_sent_total - Total alerts sent

Prometheus Configuration:

scrape_configs:
  - job_name: 'beacon'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:4000']
    metrics_path: /metrics

Grafana Dashboards

Beacon provides 5 pre-built Grafana dashboards with 33+ panels:

# List available dashboards
curl http://localhost:4000/api/grafana/dashboards

# Download specific dashboard
curl http://localhost:4000/api/grafana/dashboards/overview

Available Dashboards:

1. Overview Dashboard - System health and high-level metrics (16 panels)
2. Performance Dashboard - Detailed performance and throughput (6 panels)
3. Storage Dashboard - Storage utilization and growth tracking (6 panels)
4. Cluster Dashboard - Distributed cluster health monitoring (2 panels)
5. Alerts Dashboard - Alert system performance tracking (3 panels)

Quick Start with Monitoring Stack:

# Start Prometheus, Grafana, and AlertManager
cd examples
docker-compose -f docker-compose.monitoring.yml up -d

# Import dashboards automatically
./grafana-quickstart.sh

Access Grafana at http://localhost:3000 (admin/admin)

Alert Rules:

50+ pre-configured Prometheus alert rules across 8 categories:

• Health alerts (service down, memory issues)
• Ingestion alerts (rate changes, no logs)
• Storage alerts (growth, capacity)
• Cluster alerts (node loss, split brain)
• Real-time alerts (WebSocket issues)
• Alert system alerts (alert storms)
• Performance alerts (efficiency, utilization)
• Meta alerts (monitoring failures)

See examples/prometheus-alerts.yml for complete alert rules.

Configuration Management

View and validate runtime configuration:

# View current configuration
curl http://localhost:4000/config

# Validate configuration
curl http://localhost:4000/config/validate

Environment Variables:

Configure Beacon using environment variables:

# Server
export PORT=4000
export APPLICATION_ENV=prod

# Cluster
export NODE_NAME=node1@localhost
export REPLICATION_FACTOR=2
export SHARD_COUNT=16

# Storage
export RETENTION_DAYS=30
export STORAGE_COMPRESSION=true

# Monitoring
export METRICS_ENABLED=true
export ALERTS_ENABLED=true

See Configuration Guide for all available options.

Graceful Shutdown

Initiate graceful shutdown to ensure no data loss:

# Initiate shutdown
curl -X POST http://localhost:4000/ops/shutdown

# Check shutdown status
curl http://localhost:4000/ops/shutdown/status

The shutdown process:

1. Stops accepting new traffic
2. Notifies WebSocket clients
3. Drains the processing pipeline
4. Flushes storage
5. Cleans up cluster state

🌐 Distributed Architecture

Beacon supports distributed clustering for high availability and scalability.

Features

• Automatic Node Discovery - Using libcluster
• Data Replication - Configurable replication factor
• Consistent Hashing - For data distribution
• Load Balancing - Automatic request routing
• Fault Tolerance - Node failure handling

Cluster Management

# Get cluster status
curl http://localhost:4000/cluster/status

# Get node health
curl http://localhost:4000/cluster/nodes/health

# Trigger rebalance
curl -X POST http://localhost:4000/cluster/rebalance

# Get specific node info
curl http://localhost:4000/cluster/nodes/{node_name}

Running a Cluster

# Start multiple nodes
./start_cluster.sh

# Or manually
NODE_NAME=node1@127.0.0.1 PORT=4000 iex -S mix
NODE_NAME=node2@127.0.0.1 PORT=4001 iex -S mix
NODE_NAME=node3@127.0.0.1 PORT=4002 iex -S mix

See DISTRIBUTED.md for detailed clustering documentation.

🔍 Advanced Features

Search Capabilities

The search API offers advanced filtering and full-text search capabilities with:

• Full-text search across all log fields
• Time-range filtering
• Level-based filtering
• Source filtering
• Query optimization for time-series data
• Pagination support

Real-time Streaming

Real-time features include WebSocket streaming and subscription management with:

• Live log tailing with filtering
• Subscription-based event delivery
• Automatic reconnection handling
• Backpressure management

Alert System

Comprehensive alerting system with:

• Pattern matching and threshold-based rules
• Multiple notification channels (Webhook, Slack, Email)
• Alert state management (firing, acknowledged, resolved)
• Cooldown periods
• Alert history tracking
• Alert storm prevention

See alert examples in the Usage section above.

💻 Development

Project Structure

Beacon is organized as a monorepo with separate backend and frontend folders:

Backend (/api)

• lib/log_aggregator.ex: Main application module
• lib/log_aggregator/application.ex: OTP application setup
• lib/log_aggregator/ingestor.ex: Log ingestion system
• lib/log_aggregator/pipeline/: Log processing pipeline
• lib/log_aggregator/storage/: Storage subsystem
• lib/log_aggregator/distributed/: Clustering and distribution
• lib/log_aggregator/real_time/: Real-time subsystem
• lib/log_aggregator/alerts/: Alert management system
• lib/log_aggregator/monitoring.ex: Monitoring and metrics
• lib/log_aggregator/grafana/: Grafana dashboard templates
• lib/log_aggregator/http_endpoint.ex: HTTP API
• lib/log_aggregator/open_api/: OpenAPI/Swagger documentation

Frontend (/front)

• Modern React application with TypeScript
• Real-time dashboard for log visualization
• Search interface with advanced filtering
• Alert management UI

Examples (/examples)

• Docker Compose configurations
• Prometheus configuration and alert rules
• Grafana quickstart scripts
• Monitoring stack setup

Testing

Backend tests:

cd api
mix test

📚 Documentation

Comprehensive documentation is available:

Core Features

• Advanced search and query capabilities
• Real-time streaming and WebSocket API
• Distributed clustering architecture

Monitoring & Operations

• Complete monitoring guide (health checks, metrics)
• Grafana dashboard integration guide
• OpenAPI/Swagger documentation

Examples & Setup

• examples/README.md - Quick start guides and examples

API Documentation

• Swagger UI: http://localhost:4000/api/swagger
• OpenAPI Spec: http://localhost:4000/api/openapi

Runbooks

• Operations runbook and troubleshooting

🎯 Quick Start Scripts

# Start monitoring stack (Prometheus + Grafana)
cd examples
./grafana-quickstart.sh

📈 Performance

Beacon is designed for high performance:

• Ingestion: 100K+ logs/second per node
• Query: <100ms for recent data searches
• Memory: <2GB for 1M logs in memory
• Compression: 10:1 compression ratio for archived logs
• Clustering: Linear scalability with node count

---

Built with ❤️ using Elixir/OTP