Configure canary deployments and basic SLOs for core services

[Hyperkit-dev]

🎯 Layer 1: Intent Parsing

What needs to be done?

Task Title:

Configure canary deployments and basic SLOs for core services

Area: infra | Repos: infra/

Primary Goal:

Implement canary deployment strategy and define basic Service Level Objectives (SLOs) for core HyperAgent services to enable safe, gradual rollouts and measurable service quality

User Story / Context:

As a platform operator, I want canary deployments and basic SLOs configured so that new releases can be safely rolled out with minimal risk and service quality can be measured and maintained

Business Impact:

Reduces deployment risk and improves platform reliability. Enables data-driven deployment decisions. Critical for production readiness. Supports Phase 1 Foundation goals.

Task Metadata:

• Sprint: Sprint 3
• Related Epic/Project: GitHub Project 9 - Phase 1 Foundation
• Issue Type: Feature
• Area: Infra
• Related Documentation:
  • Platform Blueprint - Complete platform specification
  • System Architecture - Infrastructure design
  • Execution Strategy - Deployment strategy
  • Monitoring & Reporting - SLO definitions

---

📚 Layer 2: Knowledge Retrieval

What information do I need?

Required Skills / Knowledge:

• DevOps/Infra (Kubernetes, deployment strategies)
• Observability and monitoring (Prometheus, Grafana)
• SLO/SLI definitions and measurement

Estimated Effort:

M (Medium - 3-5 days)

Knowledge Resources:

• Review .cursor/skills/ for relevant patterns (devops-engineer, prometheus-configuration)
• Check .cursor/llm/docs/ for implementation examples
• Read Platform Blueprint: docs/draft.md
• Read System Architecture: docs/planning/4-System-Architecture-Design.md
• Read Execution Strategy: docs/reference/spec/execute.md
• Read Monitoring & Reporting: docs/planning/10-Monitoring-Reporting.md
• Study tech docs / ADRs in docs/adrs/ directory
• Review Kubernetes canary deployment patterns

Architecture Context:

According to the Platform Blueprint, HyperAgent uses a microservice architecture. Canary deployments will enable gradual rollout of orchestrator, agent services, and core services with automatic rollback based on SLO violations.

System Architecture Diagram:

graph TB
    subgraph "Traffic Split"
        Ingress[Ingress Controller]
        Stable[Stable Version<br/>90% Traffic]
        Canary[Canary Version<br/>10% Traffic]
    end
    
    subgraph "Monitoring"
        Metrics[Prometheus Metrics]
        SLO[SLO Evaluation]
        Alert[Alert Manager]
    end
    
    subgraph "Deployment Control"
        Rollout[Rollout Controller]
        Rollback[Auto Rollback]
    end
    
    Ingress --> Stable
    Ingress --> Canary
    Stable --> Metrics
    Canary --> Metrics
    Metrics --> SLO
    SLO --> Alert
    SLO --> Rollout
    Alert --> Rollback
    Rollout --> Ingress

Loading

Code Examples & Patterns:

Canary Deployment Example:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: hyperagent-orchestrator
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 10
      - pause: {duration: 5m}
      - setWeight: 25
      - pause: {duration: 5m}
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 100
      analysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: orchestrator

---

⚠️ Layer 3: Constraint Analysis

What constraints and dependencies exist?

Known Dependencies:

• Kubernetes cluster must be available
• Monitoring stack (Prometheus/Grafana) must be set up
• Service metrics must be instrumented (see issue Instrument services with OpenTelemetry traces and metrics #215)
• ArgoCD Rollouts or similar tool must be installed

Technical Constraints:

Scope limited to canary deployment configuration and basic SLO definitions. Full observability stack setup tracked separately.

Current Blockers:

None identified (update as work progresses)

Risk Assessment & Mitigations:

Risk of SLO violations causing unnecessary rollbacks. Mitigation: Start with conservative SLO targets, gradually tighten based on historical data, implement proper alerting thresholds.

Resource Constraints:

• Deadline: Mar 3–16 (Sprint 3)
• Effort Estimate: M (Medium - 3-5 days)

---

💡 Layer 4: Solution Generation

How should this be implemented?

Solution Approach:

Implement canary deployments using ArgoCD Rollouts or Flagger:

1. Configure Rollout resources for core services
2. Define traffic splitting strategy (10% → 25% → 50% → 100%)
3. Set up SLO definitions:
   • Availability SLO: 99.5% uptime
   • Latency SLO: p95 < 2s
   • Error rate SLO: < 0.1%
4. Configure automatic rollback on SLO violations
5. Set up SLO dashboards in Grafana

Design Considerations:

• Follow established patterns from .cursor/skills/devops-engineer
• Maintain consistency with existing deployment configurations
• Consider service-specific SLO requirements
• Ensure proper monitoring and alerting
• Plan for testing and validation
• Support gradual rollout with manual approval gates

Acceptance Criteria (Solution Validation):

• Canary deployment strategy configured for core services
• SLO definitions documented and implemented
• Automatic rollback on SLO violations working
• SLO dashboards created in Grafana
• Canary deployment tested successfully
• Documentation updated with canary deployment procedures
• Code reviewed and approved

---

📋 Layer 5: Execution Planning

What are the concrete steps?

Implementation Steps:

1. Review service requirements and define SLO targets
2. Install and configure ArgoCD Rollouts or Flagger
3. Create Rollout manifests for orchestrator service
4. Create Rollout manifests for agent services
5. Create Rollout manifests for core services
6. Configure traffic splitting strategy
7. Define SLO metrics and thresholds
8. Set up SLO evaluation and alerting
9. Create Grafana dashboards for SLO monitoring
10. Test canary deployment with rollback scenario
11. Document canary deployment procedures
12. Code review and approval

Environment Setup:
Repos / Services:

• Infra / IaC repo: hyperagent/infra/
• Kubernetes cluster with monitoring stack

Required Environment Variables:

• KUBECONFIG= (Kubernetes cluster configuration)
• PROMETHEUS_URL= (Prometheus server URL)
• GRAFANA_URL= (Grafana server URL)

Access & Credentials:

• Kubernetes cluster access: kubeconfig file
• Monitoring stack access: Internal vault
• Access request: Contact @devops or project lead

---

✅ Layer 6: Output Formatting & Validation

How do we ensure quality delivery?

Ownership & Collaboration:

• Owner: @JustineDevs
• Reviewer: @ArhonJay
• Access Request: @JustineDevs or @ArhonJay
• Deadline: Mar 3–16 (Sprint 3)
• Communication: Daily stand-up updates, GitHub issue comments

Quality Gates:

• Code follows project style guide (see .cursor/rules/rules.mdc)
• All tests pass (unit, integration, e2e)
• No critical lint/security issues
• Documentation updated (README, code comments, ADRs if needed)
• Meets all acceptance criteria from Layer 4
• Follows production standards (see .cursor/rules/production.mdc)

Review Checklist:

• Code review approved by @ArhonJay
• CI/CD pipeline passes (GitHub Actions)
• Canary deployment tested and verified
• SLO monitoring working correctly
• Security scan passes (no critical vulnerabilities)
• Documentation complete and accurate

Delivery Status:

• Initial Status: To Do
• Progress Tracking: Use issue comments for updates
• Sign-off: Approved by @Hyperionkit on [YYYY-MM-DD]
• PR Link: [Link to merged PR(s)]

Related Issues:

• Instrument services with OpenTelemetry traces and metrics #215: Instrument services with OpenTelemetry traces and metrics
• Implement release workflow (build Docker images, deploy to staging) #232: Implement release workflow

Documentation References:

• Platform Blueprint
• System Architecture
• Execution Strategy
• Monitoring & Reporting
• Project Phases