Critical Incident: reviews.default.svc.cluster.local service high error rate

[peterj]

🚨 Critical issue detected with service reviews.default.svc.cluster.local

Timestamp: recent 5-minute window

Metrics:

• Request rate: 1.06 req/s
• Error rate: 18.0% (0.19 errors/s)
• 5xx response code rate: 18.0% (503 errors)
• Latency percentiles: p50=4.3 ms, p95=14.3 ms, p99=22.9 ms

Severity: CRITICAL

Suggested investigation:

1. Check application logs for 503 errors
2. Investigate upstream/downstream service availability
3. Review recent deployments and configuration changes
4. Enable tracing if not active to track error propagation

PromQL queries for reproduction:

• Total requests: sum(rate(istio_requests_total{destination_service="reviews.default.svc.cluster.local"}[5m]))
• 5xx errors: sum(rate(istio_requests_total{destination_service="reviews.default.svc.cluster.local",response_code=~"5.."}[5m]))

Assignee: peterj

Please address this as soon as possible to restore service health.