FizzBuzz Enterprise Edition

Enterprise-grade FizzBuzz solution with event-driven architecture, AI-powered divisibility detection, and comprehensive observability.

Table of Contents

• Why FizzBuzz Enterprise Edition?
• Architecture
• Getting Started
• Services
• Analytics
• LLM Evaluations
• Testing
• Documentation
• FinOps
• Contributing
• License

Why FizzBuzz Enterprise Edition?

Modern organizations face unprecedented challenges in their FizzBuzz operations:

• Scale: Legacy FizzBuzz implementations struggle to handle more than single-digit throughput
• Accuracy: Traditional modulo operations lack the nuanced understanding required for edge cases
• Observability: Without proper instrumentation, FizzBuzz failures can go undetected for hours
• Auditability: Regulatory requirements increasingly demand full traceability of FizzBuzz determinations

FizzBuzz Enterprise Edition addresses these challenges through a cloud-native, event-driven architecture that leverages large language models for divisibility determination. Our system processes FizzBuzz requests with sub-second latency while maintaining 99.9% accuracy SLO.

Key Features

• AI-Powered Divisibility Detection: Multi-vendor LLM fallback chain (Claude → GPT → Gemini → Grok → Local → modulo) for enterprise-grade reliability
• Event-Driven Architecture: Kafka-based event streaming for reliable, scalable processing
• Comprehensive Observability: OpenTelemetry instrumentation with distributed tracing
• Modern Data Stack: dbt + DuckDB analytics pipeline for FizzBuzz business intelligence
• MCP Integration: Expose FizzBuzz as a tool for AI assistants via Model Context Protocol
• Enterprise Security: SOC 2 Type II compliant (certification pending)

Architecture

graph TB
    Client([Client]) --> Ingestion[Number Ingestion<br/>Go]
    Ingestion --> Kafka[(Kafka)]

    Kafka --> Fizz[Fizz Service]
    Kafka --> Buzz[Buzz Service]

    Fizz --> LLM1[LLM Fallback Chain]
    Buzz --> LLM2[LLM Fallback Chain]

    Fizz --> Kafka
    Buzz --> Kafka

    Kafka --> Aggregator[FizzBuzz Aggregator<br/>Windowed Join]
    Aggregator --> Kafka

    Kafka --> API[API Service<br/>GraphQL]
    API --> MCP[MCP Server]

    Kafka --> Analytics[(Analytics<br/>dbt + DuckDB)]

Loading

_{Kafka topics in Redpanda Console}

Data Flow

1. Ingestion: Numbers enter the system via the Go-based Number Ingestion Service (chosen for its superior performance characteristics; see ADR-027)
2. Fan-out: Numbers are published to number-topic where both Fizz and Buzz services consume them
3. AI Determination: Each service uses the LLM fallback chain to determine divisibility (3 for Fizz, 5 for Buzz)
4. Aggregation: The FizzBuzz Aggregator performs a windowed stream join on fizz-topic and buzz-topic
5. Result Publishing: Final FizzBuzz results are emitted to fizzbuzz-topic
6. Serving: The GraphQL API serves results to clients

Getting Started

Prerequisites

• Docker & Docker Compose (recommended for local development)
• An Anthropic API key

For running services outside Docker:

• Node.js 20+
• Go 1.21+
• Python 3.11+
• An Upstash account (free tier available)

Quick Start

git clone https://github.com/joshuaisaact/fizzbuzz-enterprise-edition-2026.git
cd fizzbuzz-enterprise-edition-2026
cp .env.example .env
# Add your Anthropic API key to .env

docker compose up -d

For running outside Docker (requires Upstash):

pnpm install
cd services/number-ingestion-service && go mod download && cd ../..

# Start services (each in separate terminal)
pnpm dev:fizz
pnpm dev:buzz
pnpm dev:aggregator
pnpm dev:api
cd services/number-ingestion-service && go run .

Local Development (Docker)

For local development without Upstash, use Docker Compose with Redpanda (Kafka-compatible):

# Copy environment template
cp .env.example .env
# Add your Anthropic API key to .env

# Start all services
docker compose up -d

# View logs
docker compose logs -f

# Access services:
# - Number Ingestion API: http://localhost:8080
# - GraphQL API: http://localhost:4000
# - Redpanda Console: http://localhost:8888
# - Schema Registry: http://localhost:18081
# - Grafana: http://localhost:3000 (admin/fizzbuzz)
# - Prometheus: http://localhost:9090
# - OTel Collector (metrics): http://localhost:8889/metrics

# Stop services
docker compose down -v

Verify Installation

# Submit a number for FizzBuzz processing
curl -X POST http://localhost:8080/api/v1/numbers \
  -H "Content-Type: application/json" \
  -d '{"value": 15}'

# Query results via GraphQL
curl -X POST http://localhost:4000/graphql \
  -H "Content-Type: application/json" \
  -d '{"query": "{ fizzbuzz(number: 15) { result confidence latencyMs } }"}'

Services

Service	Language	Description	Port
number-ingestion-service	Go	High-performance number ingestion	8080
fizz-service	TypeScript	AI-powered divisibility-by-3 detection	-
buzz-service	TypeScript	AI-powered divisibility-by-5 detection	-
fizzbuzz-aggregator	TypeScript	Stream join and result aggregation	-
api-service	TypeScript	GraphQL API gateway	4000
local-llm	Ollama	Local LLM inference (OpenAI-compatible)	11434
mcp-server	TypeScript	MCP tool server for AI assistants	stdio

MCP Integration

FizzBuzz Enterprise Edition exposes FizzBuzz capabilities to AI assistants via the Model Context Protocol.

Available Tools

• fizzbuzz - Get FizzBuzz result for a specific number with metadata (latency, cost, correlation ID)
• recent-results - Retrieve recent FizzBuzz computations
• stats - Get store statistics (total results, oldest entry age)

Claude Desktop Configuration

Add to your claude_desktop_config.json:

{
  "mcpServers": {
    "fizzbuzz": {
      "command": "node",
      "args": ["/path/to/fizzbuzz-enterprise-edition-2026/mcp-server/dist/index.js"],
      "env": {
        "API_SERVICE_HOST": "localhost",
        "API_SERVICE_PORT": "4000"
      }
    }
  }
}

Example Interaction

User: What's the FizzBuzz result for 15?

Claude: [Calls fizzbuzz tool with number=15]

The result for 15 is "fizzbuzz" - it's divisible by both 3 and 5.

Processing metadata:
- Latency: 142ms
- Cost: $0.00023

See mcp-server/README.md for full documentation.

Analytics

FizzBuzz Enterprise Edition includes a complete modern data stack for FizzBuzz analytics:

Events (S3/Local) → dbt Models → DuckDB → Dashboards

dbt Models

• stg_fizzbuzz_events.sql - Staged raw events
• int_fizzbuzz_classified.sql - Intermediate classification logic
• fct_fizzbuzz_results.sql - Fact table of FizzBuzz results
• dim_divisibility.sql - Divisibility dimension table

_{dbt lineage graph}

See analytics/README.md for details.

LLM Evaluations

Following INC-2025-1114-001 (Fizz returned true for 7), we implemented continuous LLM accuracy evaluation using promptfoo.

# Run all evaluations
pnpm eval

# Run service-specific evaluations
pnpm eval:fizz    # Divisibility by 3
pnpm eval:buzz    # Divisibility by 5

# View results in browser
pnpm eval:view

Test Coverage

• 85+ test cases across Fizz and Buzz services
• Golden datasets with version-controlled expected outputs
• Regression tests for all historical incidents
• Cultural number tests to catch LLM semantic confusion (42, 666, 777, etc.)

Model Comparison

Current production model (claude-3-haiku) achieves 100% accuracy on both Fizz and Buzz evaluations. Historical comparison data is available in packages/evals/results/.

_{promptfoo evaluation results}

See packages/evals/README.md for full documentation.

Testing

FizzBuzz Enterprise Edition uses a multi-layer testing strategy:

Layer	Command	Description
Unit Tests	pnpm test	Fast, isolated service tests with mocked dependencies
Integration Tests	pnpm test:integration	Cross-service contract verification with Testcontainers
E2E Tests	docker compose up	Full pipeline validation

Integration Tests

Integration tests verify message contracts between services using Redpanda (Kafka-compatible) in Testcontainers:

# Run integration tests
pnpm test:integration

See tests/README.md for details on writing and running integration tests.

Documentation

Architecture Decision Records

ADR	Title	Status
ADR-001	Why we chose event-driven architecture	Accepted
ADR-002	Schema Registry for event contracts	Accepted
ADR-019	Migration from monolith to microservices	Accepted
ADR-024	Reverting ADR-019	Rejected
ADR-027	Go for ingestion service	Accepted
ADR-031	Post-mortem: Fizz returned true for 7	Accepted
ADR-038	Prompt caching cost analysis	Accepted
ADR-041	Why we are not returning to a monolith	Accepted
ADR-042	OpenTelemetry adoption for enterprise observability	Accepted
ADR-047	LLM evaluation strategy with promptfoo	Accepted
ADR-048	MCP server implementation	Accepted
ADR-049	Multi-vendor LLM provider fallback chain	Accepted

Runbooks

• Fizz Service Not Fizzing
• Buzz Latency Exceeded
• Kafka Consumer Lag
• Aggregator Service Issues

On-Call

• Rotation Policy - Schedules, responsibilities, and compensation
• Incident Severity Matrix - P1-P4 definitions and response expectations
• War Room Protocol - Major incident coordination procedures
• Post-Incident Review Template - Template for incident retrospectives

Incident Retrospectives

• 2025-11-14: Fizz Returned True for 7

FinOps

FizzBuzz Enterprise Edition includes comprehensive cost tracking for LLM operations.

Current Metrics (30-day rolling average)

Metric	Value
Cost per Fizz	$0.00012
Cost per Buzz	$0.00011
Cost per FizzBuzz	$0.00023
Daily burn rate	$2.47
Monthly projected	$74.10

Cost Optimization

• Prompt caching reduces costs by ~40% (see ADR-038)
• Haiku model provides 90% cost savings vs Sonnet with comparable accuracy
• Batch processing amortizes fixed costs across multiple numbers

See docs/finops/ for detailed cost analysis and optimization strategies.

Local LLM (Air-Gapped Deployments)

For environments without internet access or where cloud API dependencies are unacceptable, FizzBuzz Enterprise Edition includes local LLM inference via Ollama.

Architecture

The local provider slots into the fallback chain between xAI Grok and the mathematical Modulo fallback:

Claude → GPT → Gemini → Grok → Local (Ollama) → Modulo

See ADR-050 for detailed evaluation of deployment options.

Configuration

# Environment variables (all optional - sensible defaults provided)
LOCAL_LLM_BASE_URL=http://localhost:11434/v1  # Ollama endpoint
LOCAL_LLM_MODEL=qwen2.5:1.5b                  # Model selection
LOCAL_LLM_TIMEOUT_MS=60000                    # Timeout for inference
LOCAL_LLM_COST_PER_1M_TOKENS=0.01             # Internal chargeback rate
LLM_FALLBACK_LOCAL=true                       # Enable/disable in chain

Running Locally

The local LLM service is included in Docker Compose:

docker compose up local-llm -d
docker compose exec local-llm ollama pull qwen2.5:1.5b
docker compose up fizz-service buzz-service -d

For standalone usage (Ollama installed locally):

ollama serve &
ollama pull qwen2.5:1.5b

Use Cases

• Air-gapped environments: Government, healthcare, financial services
• Local development: No API keys needed for dev/test
• Edge deployments: Low-latency on-premise inference
• Cost optimization: High-volume scenarios with amortized hardware costs

Observability

All services are instrumented with OpenTelemetry. Metrics, traces, and logs are exported via OTLP to the OpenTelemetry Collector.

Local Telemetry Stack

The Docker Compose setup includes a complete observability pipeline:

Services → OTLP → OpenTelemetry Collector → Prometheus → Grafana

Component	Port	Description
OTel Collector (OTLP gRPC)	4317	Receives telemetry from services
OTel Collector (OTLP HTTP)	4318	Receives telemetry from services
OTel Collector (Prometheus)	8889	Exposes metrics for Prometheus scraping
Prometheus	9090	Metrics storage and querying
Alertmanager	9093	Alert routing and notification
Grafana	3000	Dashboards and visualization

Access Grafana at http://localhost:3000 (login: admin/fizzbuzz) to view dashboards. Access Prometheus at http://localhost:9090 for direct queries. Example PromQL:

# Total messages processed by the aggregator
fizzbuzz_aggregator_messages_aggregated_total

# Fizz service processing time (95th percentile)
histogram_quantile(0.95, sum(rate(fizzbuzz_fizz_message_processing_duration_ms_bucket[5m])) by (le))

# Messages per second by service
sum(rate(fizzbuzz_fizz_messages_processed_total[1m]))

Dashboards

Pre-built Grafana dashboards are automatically provisioned:

• FizzBuzz Overview - Executive summary with SLO metrics, throughput, and cost tracking
• Service Operations - Detailed per-service metrics for on-call engineers

Overview Dashboard

Service Operations

Key Metrics

• fizzbuzz.fizz.throughput - Fizz determinations per second
• fizzbuzz.buzz.throughput - Buzz determinations per second
• fizzbuzz.fizz.latency_p99 - 99th percentile Fizz latency
• fizzbuzz.buzz.latency_p99 - 99th percentile Buzz latency
• fizzbuzz.aggregator.join_window_size - Current window size for stream join
• fizzbuzz.llm.tokens_used - Total tokens consumed
• fizzbuzz.llm.cost_usd - Running cost in USD

Kubernetes Deployment

For production deployment, use the Kustomize manifests in k8s/:

# Create namespace and secret
kubectl create namespace fizzbuzz
kubectl create secret generic anthropic-credentials \
  --namespace=fizzbuzz \
  --from-literal=ANTHROPIC_API_KEY=sk-ant-...

# Deploy to staging (single replica, debug logging)
kubectl apply -k k8s/overlays/staging

# Deploy to production (HA, ingress, PDBs)
kubectl apply -k k8s/overlays/production

See k8s/README.md for details.

Contributing

Please read CONTRIBUTING.md before submitting changes.

Requirements:

• DCO sign-off on all commits
• Conventional commit messages
• 3 approving reviews from CODEOWNERS
• All CI checks passing
• No decrease in test coverage

Security

For security vulnerabilities, please see SECURITY.md for our responsible disclosure process.

License

FizzBuzz Enterprise Edition is available under the following license tiers:

Tier	Numbers/Month	Support	Price
Community	100	GitHub Issues	Free
Professional	10,000	Email (48h SLA)	$499/mo
Enterprise	Unlimited	Dedicated CSM, 4h SLA	Contact Sales
Government	Unlimited	FedRAMP, dedicated instance	Contact Sales

See LICENSE.md for full terms.

---

_{Built with ❤️ by the FizzBuzz Platform Team}