Agent Performance Report - Week of 2026-03-13

[github-actions[bot]]

Executive Summary

• Agents analyzed: 36 workflows (25 runs in last 2 days, 80 runs over 7 days)
• Total outputs reviewed: 24 completed runs (17 success, 7 failure) — 2-day window
• Overall success rate: 71% raw / 94% adjusted (removing known lockdown failures)
• Quality score: 86/100 (↑1 from last week)
• Effectiveness score: 86/100 (↑1 from last week)
• Ecosystem health: 72/100 (stable — P1 lockdown persists, no new regressions)
• Top performers: AI Moderator, Daily Safe Outputs Conformance Checker, Semantic Function Refactoring
• Needs attention: Issue Monster (lockdown), Daily Copilot PR Merged Report (new failure), Smoke tests (intermittent)

---

Performance Rankings

🏆 Top Performing Agents

Rank	Agent	Success Rate	Avg Duration	Notes
1	AI Moderator	100% (3/3)	9.5m	Consistent, fully recovered
2	Daily Safe Outputs Conformance Checker	100%	4.2m	High quality outputs daily
3	Semantic Function Refactoring	100%	7.7m	High value, 2M tokens
4	Lockfile Statistics Analysis Agent	100%	7.4m	Reliable, consistent
5	The Great Escapi	100% (last 2d)	3.7m	Efficient, 77K tokens
6	Auto-Triage Issues	100% (2/2)	3.7m	Fast, effective triaging
7	Daily Documentation Updater	100%	8.1m	Highest token user ($1.41/run)
8	Chroma Issue Indexer	100%	5.4m	High volume, 1.7M tokens
9	Slide Deck Maintainer	100%	6.2m	Stable
10	Daily Team Evolution Insights	100%	6.7m	Good engagement

📉 Agents Needing Improvement

1. Issue Monster — Quality: 20/100, Effectiveness: 0/100

• 6/6 runs failed in last 2 days (100% failure rate)
• Root cause: GH_AW_GITHUB_TOKEN missing ([P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315 — P0 unresolved)
• No fix path currently available; all closed issues ([P1] Lockdown mode failing: GH_AW_GITHUB_TOKEN not configured — 5 workflows affected #17414, [q] fix(workflows): remove explicit lockdown:true to stop recurring failures #17807 both closed "not_planned")
• Action: Awaiting manual intervention on [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315

2. Daily Copilot PR Merged Report — ⚠️ New regression today

• Failed today's run (1,686,703 tokens consumed, error in final step)
• Previously working; needs investigation
• Recommend: Audit run §23062733795

3. Smoke Create Cross-Repo PR — Quality: 30/100

• 100% failure in 7-day window ([aw] Smoke Update Cross-Repo PR failed #20288 open)
• Cross-repo permission configuration issue

4. Smoke Copilot (scheduled) — Quality: 40/100

• 0% success in 7-day scheduled runs (event-triggered runs succeed)
• Intermittent permission issues with cross-repo operations

⏸️ Inactive / Skipped Agents (7-day window)

61 of 80 runs in the 7-day window were skipped — these are event-triggered workflows (PRs, comments, slash commands) with no qualifying events during the period. This is expected behavior, not a problem.

Notable event-triggered agents with no recent activity:

• Archie, Q, PR Nitpick Reviewer, Scout, /cloclo — all event-triggered, no issues

---

Quality Analysis

Output Quality Distribution

• Excellent (80-100): 12 agents — consistent delivery
• Good (60-79): 8 agents — minor issues only
• Fair (40-59): 3 agents — smoke test failures (infra, not agent quality)
• Poor (<40): 2 agents — Issue Monster (token), Daily Copilot PR Merged Report (new)

Common Quality Issues

1. Token lockdown (P0): Issue Monster unable to operate — affects 4 workflows total
2. Cross-repo permissions: Smoke Create Cross-Repo PR + Smoke Update Cross-Repo PR
3. New regression: Daily Copilot PR Merged Report — unknown cause

Cost & Resource Analysis

Today's runs — total cost: $4.28, 11.8M tokens

Agent	Tokens	Cost
Daily Documentation Updater	3,044,750	$1.414
Semantic Function Refactoring	2,022,292	$1.140
Lockfile Statistics Analysis Agent	946,848	$0.837
Daily Team Evolution Insights	558,139	$0.623
Daily Safe Outputs Conformance Checker	285,584	$0.269
Slide Deck Maintainer	462,049	—
Chroma Issue Indexer	1,690,064	—
Daily Copilot PR Merged Report	1,686,703	—
Daily Repository Chronicle	668,567	—
Auto-Triage Issues	~206K (2x)	—
Contribution Check	188,403	—
The Great Escapi	77,339	—

Efficiency observations:

• Daily Documentation Updater is the most expensive agent at $1.41/run — monitor for value delivered
• Semantic Function Refactoring at $1.14/run represents significant investment — verify PR outcomes
• Chroma Issue Indexer and Daily Copilot PR Merged Report consume 1.7M tokens each with $0 recorded cost (billing not captured)

7-Day Trend Analysis (Mar 7–13)

Runs breakdown:

• Total: 80 runs
• Skipped: 61 (76% — normal for event-triggered workflows)
• Completed: 19 non-trivial runs
• Success rate (non-trivial): ~63%

Recurring failures this week:

• Issue Monster: persistent GH_AW_GITHUB_TOKEN ([P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315)
• Smoke Copilot: 0% on scheduled runs
• Smoke Claude: 50% (1 failure, 1 success)
• Smoke Create Cross-Repo PR: 100% failure

Recoveries this week (major positive):

• ✅ Smoke Codex: RECOVERED after 2-week OpenAI restriction
• ✅ Duplicate Code Detector: RECOVERED
• ✅ AI Moderator: Fully healthy (after earlier Codex issues)
• ✅ Contribution Check: Recovered from 3/11 failure

Engine distribution:

• Copilot: ~70% of runs
• Claude: ~15%
• Codex: ~10%
• Gemini: ~5%

---

Behavioral Patterns

✅ Productive Patterns

• Daily orchestrators running in parallel — Documentation Updater, Team Evolution, Repository Chronicle all complete within ~8m windows efficiently
• Auto-Triage + Issue Monster coordination — When Issue Monster recovers, triage pipeline is ready
• AI Moderator consistency — Provides stable moderation across all 3 daily runs
• Semantic Refactoring depth — 56 turns, 2M tokens indicates thorough analysis, not shallow pass

⚠️ Problematic Patterns

• Issue Monster failure cascade — 5 failures per day × indefinite = accumulated workflow debt
• Smoke test infra fragility — Cross-repo smoke tests fail on permissions more than agent logic
• Cost concentration — Top 4 agents consume 85%+ of daily token budget
• Zero safe_output_count for most runs — metrics capture may be incomplete (expected for many workflows)

---

Recommendations

🔴 High Priority

1. Escalate GH_AW_GITHUB_TOKEN (#20315)

• Issue Monster, PR Triage, Daily Issues Report, Org Health all blocked
• 4 workflows × daily failures = ~28 wasted runs/week
• Requires GitHub org admin to provision token or alternative auth approach
• No current fix path — needs human escalation beyond standard issue process

2. Investigate Daily Copilot PR Merged Report regression

• New failure today after previous successes
• 1.7M tokens consumed but run failed — potential wasted cost
• Audit job logs at run #23062733795

3. Smoke test infrastructure review

• Smoke Create Cross-Repo PR ([aw] Smoke Update Cross-Repo PR failed #20288) and Smoke Update Cross-Repo PR ([aw] Smoke Update Cross-Repo PR failed #20288) both failing
• Root cause likely shared: cross-repo permission setup, not agent prompts
• A single infrastructure fix could resolve multiple smoke test failures

🟡 Medium Priority

4. Daily Documentation Updater cost monitoring

• $1.41/run × ~30 days = ~$42/month
• Establish baseline: is the cost justified by documentation improvements?
• Consider token budget caps

5. Add Smoke Gemini tracking issue

• Currently failing with add_comment context error
• No tracking issue exists (noted in previous health report)
• Impact: Gemini engine coverage gap

6. Contribution Check flakiness

• Failed 3/11, succeeded 3/13 (56 turns → high effort run)
• The 56-turn run suggests it's working hard but possibly retrying
• Investigate if there's a reliability issue or expected variance

🟢 Low Priority

7. Standardize safe_output tracking — Many workflows show safe_output_count=0 in logs even when producing outputs; metric capture may need improvement for better future analysis

8. Review event-triggered agent activation rates — 76% skip rate is healthy but audit to ensure no missed events

---

Trends

Metric	This Week	Last Week	Trend
Quality score	86/100	85/100	↑ +1
Effectiveness	86/100	85/100	↑ +1
Ecosystem health	72/100	72/100	→ stable
P1 failures	4 workflows	4 workflows	→ unchanged
Recoveries	+2 (Codex, Dup Detector)	—	✅ improving
Total active workflows	166	166	→ stable
Daily cost (est.)	~$4.28	~$4.00	↑ slight increase

Actions Taken This Run

• Generated this performance report discussion
• Confirmed Smoke Codex recovery (issues [aw] Smoke Codex failed #20285, [aw] Duplicate Code Detector failed #20304 can be closed)
• Identified Daily Copilot PR Merged Report as new regression to investigate
• Updated shared memory with current state

Next Steps

1. Human escalation needed on [P1] Lockdown token failures: Issue Monster, PR Triage Agent, Daily Issues Report #20315 (GH_AW_GITHUB_TOKEN) — no automated fix path
2. Audit Daily Copilot PR Merged Report failure (run #23062733795)
3. Close [aw] Smoke Codex failed #20285 (Smoke Codex) and [aw] Duplicate Code Detector failed #20304 (Duplicate Code Detector) — both recovered
4. Create tracking issue for Smoke Gemini failures
5. Monitor Semantic Function Refactoring and Documentation Updater cost trend

---

Analysis period: 2026-03-07 to 2026-03-13
Next report: 2026-03-20

References:

• §23062733795 — This run (Agent Performance Analyzer)
• §22941596501 — Workflow Health Monitor (2026-03-11)
• §23015353310 — Agent Performance (2026-03-12, previous run)

AI generated by Agent Performance Analyzer - Meta-Orchestrator · history

• expires on Mar 14, 2026, 5:43 PM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-14T17:43:02.033Z.

Closed by Workflow