🚨 ArgoCD Deployment Failed: agentic-platform-engineering-demo

[github-actions]

ArgoCD Deployment Failure

Application: agentic-platform-engineering-demo
Timestamp: 2026-02-17T06:59:14.583Z

Cluster Information

Field	Value
Cluster Name	msftgbb
Resource Group	agentic-platform-engineering
Namespace	default

Application Status

Field	Value
Health Status	Degraded
Sync Status	Synced
Revision	93cda02107dcb0d9ff7fa90632afbcc1fa63fd92
Repository	https://github.com/microsoftgbb/agentic-platform-engineering.git

Error Message

successfully synced (all tasks run)

Troubleshooting Commands

# Check application status in ArgoCD
argocd app get agentic-platform-engineering-demo

# Check pods in namespace
kubectl get pods -n default

# Describe failed pods
kubectl describe pods -n default

# Get pod logs
kubectl logs -n default <pod-name>

# Check events
kubectl get events -n default --sort-by='.lastTimestamp'

Quick Links

• ArgoCD UI
• Source Repository

---

This issue was automatically created by ArgoCD Notifications

[github-actions]

🔍 Cluster Doctor Diagnostic Report

Executive Summary

Status: ✅ Root cause identified and fix branch created
Severity: Medium - Application degraded but not completely down
Cluster Identity Confirmed: ✅ Yes (msftgbb/agentic-platform-engineering)

---

Root Cause Analysis

Issue Identified

The deployment is failing due to a typo in the container image name in the order-service deployment manifest.

Location: Act-3/argocd/apps/broken-aks-store-all-in-one.yaml:196

• ❌ Current (Incorrect): ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0
• ✅ Should be: ghcr.io/azure-samples/aks-store-demo/order-service:2.1.0
• Error: Missing 'c' in 'service' → typed as 'servie'

Evidence Collected

1. Cluster Verification (Identity Certainty ✅)

Cluster Name: msftgbb
Resource Group: agentic-platform-engineering  
Region: canadacentral
Kubernetes Version: 1.33.6
Network Plugin: azure (overlay mode)
Network Policy: calico
Status: Running

2. Pod Status

NAME                                READY   STATUS             RESTARTS   AGE
order-service-77bf896788-qhm4r      0/1     ImagePullBackOff   0          5m
order-service-575df9db99-gwddb      1/1     Running            0          81m   (old version)

3. Error Details from Pod Events

Failed to pull image "ghcr.io/azure-samples/aks-store-demo/order-servie:2.1.0": 
failed to resolve reference: failed to authorize: 
failed to fetch anonymous token: 403 Forbidden

The 403 error occurs because the image order-servie does not exist in the container registry. The correct image name is order-service.

4. Init Container Status

• ✅ Init container wait-for-rabbitmq completed successfully
• ❌ Main container order-service stuck in ImagePullBackOff

5. Deployment Status

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
order-service      1/1     1            1           93m

Note: Shows 1/1 ready because the old replica set (575df9db99) is still running with the correct image.

---

Impact Assessment

Current Impact:

• Application health status: Degraded (not Critical)
• Old version continues to run and serve traffic
• New deployment cannot roll out due to image pull failure
• ArgoCD shows Synced but Degraded

No Impact On:

• Running application functionality (old version still serving)
• Other microservices (all healthy)
• Cluster control plane
• Network connectivity

---

Remediation

Fix Applied

A fix has been prepared in branch: fix/cluster-doctor-issue-9-typo-order-service-image

Changes:

• File: Act-3/argocd/apps/broken-aks-store-all-in-one.yaml
• Line: 196
• Change: order-servie → order-service

Branch Details:

• Branch: fix/cluster-doctor-issue-9-typo-order-service-image
• Commit: 0be554e
• Status: Pushed to origin

Manual PR Creation Required

⚠️ Note: Automated PR creation failed due to GitHub Actions permissions. Please create the PR manually from the branch:

https://github.com/microsoftgbb/agentic-platform-engineering/compare/fix/cluster-doctor-issue-9-typo-order-service-image

Post-Fix Validation Steps

Once the PR is merged and ArgoCD syncs:

1. Verify Pod Status:

   kubectl get pods -n default -l app=order-service -o wide

   Expected: New pod in Running state with 1/1 ready

2. Check Events:

   kubectl get events -n default --field-selector involvedObject.name=order-service-77bf896788-<pod-id>

   Expected: Successful image pull and container start events

3. Verify Deployment:

   kubectl rollout status deployment/order-service -n default

   Expected: deployment "order-service" successfully rolled out

4. Confirm ArgoCD Health:

   argocd app get agentic-platform-engineering-demo

   Expected: Health Status changes from Degraded to Healthy

---

Rollback Plan

If issues occur after applying the fix:

Option 1: ArgoCD Rollback

argocd app rollback agentic-platform-engineering-demo

Option 2: Kubectl Rollback

kubectl rollout undo deployment/order-service -n default

Option 3: Manual Scale

# Scale down the problematic replica set
kubectl scale deployment order-service --replicas=0 -n default
kubectl scale deployment order-service --replicas=1 -n default

---

Additional Observations

Cluster Health: ✅ Healthy

• All other 9 pods running normally
• No resource pressure (CPU/Memory)
• Autoscaler configured and operational
• Network policies: Calico active

No Security Issues Detected:

• RBAC properly configured (Azure RBAC enabled)
• Workload Identity enabled
• Image Cleaner enabled (7-day interval)
• TLS properly configured

No Network Issues:

• All service-to-service communication working
• RabbitMQ connectivity verified (init container succeeded)
• MongoDB connectivity healthy

---

Confidence Level

Diagnosis Confidence: 🟢 Very High (95%)

• Direct evidence from pod describe and events
• Clear error message pointing to non-existent image
• Typo confirmed by examining manifest file
• All other components healthy

Fix Confidence: 🟢 Very High (98%)

• Single character correction
• No side effects expected
• Image name verified against Azure Samples repo conventions
• Surgical change with clear rollback path

---

Audit Trail

• Diagnostics Started: 2026-02-17T07:01:32Z
• Cluster Identity Verified: ✅ 2026-02-17T07:02:15Z
• Root Cause Identified: 2026-02-17T07:03:45Z
• Fix Branch Created: 2026-02-17T07:04:30Z
• Fix Branch Pushed: 2026-02-17T07:05:15Z
• Actions Taken: Read-only diagnostics only (no write operations to cluster)

---

Next Steps

1. ✅ Completed: Diagnosis and fix preparation
2. ⏳ Pending: Manual PR creation (due to GH Actions permissions)
3. ⏳ Pending: PR review and merge
4. ⏳ Pending: ArgoCD sync
5. ⏳ Pending: Post-deployment validation

Recommended Action: Create and merge PR from branch fix/cluster-doctor-issue-9-typo-order-service-image to resolve this issue.

---

Report generated by Cluster Doctor Agent | Safe Mode: Read-Only Diagnostics