🚀 Nginx Monitoring with OpenTelemetry, Prometheus & Grafana

This guide provides step-by-step instructions to set up Nginx monitoring using OpenTelemetry with a separate collector server, sending metrics to Prometheus, and visualizing them in Grafana.

---

1️⃣ Architecture Overview

Nginx (Monolithic)
     │
     ├──► OpenTelemetry Collector (Separate Server)
     │         ├──► Prometheus (Metrics Storage)
     │         ├──► Grafana (Visualization - Dashboard UID: 14282)

---

2️⃣ Install OpenTelemetry Collector on a Separate Server

SSH into your dedicated OpenTelemetry Collector server and install the OpenTelemetry Collector.

mkdir -p /etc/otel && cd /etc/otel
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/latest/download/otelcol-linux-amd64
chmod +x otelcol-linux-amd64
mv otelcol-linux-amd64 /usr/local/bin/otelcol

Create OpenTelemetry Configuration File

nano /etc/otel/otel-config.yaml

Add the following:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  nginx:
    endpoint: "http://your-nginx-server/nginx_status"

exporters:
  prometheus:
    endpoint: "0.0.0.0:9464"

  logging:
    loglevel: debug

service:
  pipelines:
    metrics:
      receivers: [nginx]
      exporters: [prometheus]

• Replace your-nginx-server with the actual IP or hostname of your Nginx server.

Run OpenTelemetry Collector

otelcol --config /etc/otel/otel-config.yaml > /var/log/otelcol.log 2>&1 &

OR set it as a systemd service:

nano /etc/systemd/system/otel-collector.service

Add:

[Unit]
Description=OpenTelemetry Collector
After=network.target

[Service]
ExecStart=/usr/local/bin/otelcol --config /etc/otel/otel-config.yaml
Restart=always

[Install]
WantedBy=multi-user.target

Enable and start the service:

systemctl daemon-reload
systemctl enable otel-collector
systemctl start otel-collector

---

3️⃣ Configure Nginx to Send Data to OpenTelemetry Collector

On your Nginx server, install the OpenTelemetry module and configure it.

Install OpenTelemetry Module for Nginx

sudo apt-get install nginx-opentelemetry-module

Modify Nginx Configuration

nano /etc/nginx/nginx.conf

Add:

load_module modules/ngx_http_opentelemetry_module.so;

http {
    opentelemetry_config /etc/nginx/otel.yaml;
    
    server {
        listen 80;
        location / {
            opentelemetry_trace_context;
        }
    }
}

Create OpenTelemetry Configuration File

nano /etc/nginx/otel.yaml

Add:

collector_endpoint: "http://otel-collector-server:4317"
service_name: "nginx-monolith"
sample_rate: 1.0

• Replace otel-collector-server with the IP/hostname of your OpenTelemetry Collector server.

Restart Nginx

systemctl restart nginx

---

4️⃣ Configure Prometheus to Scrape OpenTelemetry Collector

Now, on your Prometheus server, edit the prometheus.yml file.

nano /etc/prometheus/prometheus.yml

Add:

scrape_configs:
  - job_name: 'otel-nginx'
    static_configs:
      - targets: ['otel-collector-server:9464']

Replace otel-collector-server with the actual IP of your OpenTelemetry Collector server.

Restart Prometheus:

systemctl restart prometheus

---

5️⃣ Import Grafana Dashboard

Now that OpenTelemetry is collecting metrics and sending them to Prometheus, visualize them in Grafana.

1. Open Grafana → Dashboards → Import.
2. Enter UID: 14282 (Pre-built OpenTelemetry Dashboard).
3. Select Prometheus as the data source.
4. Click Import.

---

6️⃣ Verify Everything

Check if OpenTelemetry Collector is receiving metrics

On the OpenTelemetry Collector server, run:

curl http://localhost:9464/metrics | grep nginx

Expected output:

otel_http_server_duration_seconds{host="nginx-monolith"} 0.5
otel_http_server_errors_total{host="nginx-monolith", status="500"} 2

Check Prometheus

1. Open Prometheus UI:

   http://prometheus-server:9090
   

2. Run a query for:

   otel_http_server_duration_seconds
   

Check Grafana

1. Open Grafana.
2. Check if Dashboard UID 14282 is showing real-time Nginx metrics.

---

7️⃣ Key Metrics Monitored

Metric	Description
otel_http_server_duration_seconds	Time taken to process HTTP requests
otel_http_server_errors_total	Count of HTTP errors (4xx, 5xx)
otel_http_upstream_latency_seconds	Latency in responses from upstream servers
otel_http_request_size_bytes	Size of incoming HTTP requests
otel_http_response_size_bytes	Size of outgoing responses
otel_http_request_count_total	Total number of processed requests
nginx_active_connections	Number of active client connections
nginx_waiting_connections	Number of idle client connections
otel_trace_id	Unique trace ID for distributed tracing

---

✅ Final Steps

1. Run:

   curl http://otel-collector-server:9464/metrics | grep nginx

   If output appears, OpenTelemetry is correctly collecting Nginx metrics.

2. Check Prometheus (http://prometheus-server:9090) for otel_http_server_duration_seconds.

3. Go to Grafana and view Dashboard UID 14282 for real-time Nginx metrics.

---

🎯 Conclusion

With this setup, you now have:

• OpenTelemetry Collector (separate server) collecting Nginx metrics
• Prometheus storing metrics
• Grafana visualizing them (Dashboard UID 14282)
• Distributed tracing, structured logs, and advanced monitoring

🚀 Enjoy powerful Nginx monitoring with OpenTelemetry! 🚀