Audit Report: Agentic Workflows — 2026-03-25

[github-actions[bot]]

Summary

Daily audit of agentic workflow runs for 2026-03-25. The logs cache contains 163 run entries; 17 had full summary/metadata available for analysis. The GitHub API rate limit was hit during live log fetching, so analysis relied on the pre-downloaded log cache.

Metric	Value
Runs analyzed (with data)	17
Engines active	Claude, Copilot, Codex
Failures	1 (Daily Workflow Updater)
Cancelled	4 (PR-triggered CI batch)
Missing tool reports	1 (Serena MCP)
Total tokens (known)	~339,349 (Daily Workflow Updater)

---

Workflow Health

All three engines (Claude, Copilot, Codex) were active today. Most runs were triggered by PR #22962. The primary failure was in the Daily Workflow Updater (Copilot) at run §23563694023, where the threat detection job failed. All other runs that completed were successful or completed their work (smoke tests, changeset checks, agent container tests).

---

Token & Cost Analysis

Token usage data is sparse in the run summaries due to many runs still being in progress at collection time. The Daily Workflow Updater consumed 339,349 tokens across 9 turns (stored in metrics.TokenUsage but not propagated to run.TokenUsage). Most other workflows (smoke tests, changeset generators) completed in 0–2 turns with minimal token use.

---

Critical Findings

🔴 Daily Workflow Updater — Detection Job Failure

• Run: §23563694023
• Root cause: The Copilot CLI failed to authenticate during threat detection (exit code 1). The detection log contained 119 lines but no THREAT_DETECTION_RESULT JSON, causing the parse step to error out.
• Impact: The safe_outputs job was skipped due to detection conclusion being failure, but the agent job itself succeeded — the workflow created a PR updating setup action from v0.63.1 → v0.64.0.
• Recommendation: Investigate Copilot CLI token availability/permissions in the detection job container. Consider adding a fallback or retry mechanism.

⚠️ Smoke Copilot — Poor Agentic Control Assessment

• Run: §23563669474
• Assessments: resource_heavy_for_domain (medium) and poor_agentic_control (high)
• Evidence: 12 write actions, high actuation friction. The smoke test performed PR reviews, created issues, dispatched workflows, posted comments, created a discussion — broad scope for a smoke test.
• Recommendation: Consider scoping smoke tests more tightly. Separate "capability tests" (proving tools work) from "production actions" (creating real issues, reviews, discussions).

🟡 Serena MCP Not Available

• Run: §23563669474 (Smoke Copilot)
• Serena MCP server (activate_project, find_symbol) is not registered in this environment. The smoke test reports this as a partial pass.
• This appears to be a known/expected gap — the smoke test already handles it gracefully.

ℹ️ 4 Cancelled Runs (PR #22962 CI Batch)

• Runs 23563669394, 23563669404, 23563669407, 23563669426 were all cancelled.
• These were triggered on refs/pull/22962/merge — a typical pattern when a PR is updated/force-pushed, cancelling the previous CI batch. Not a concern.

---

Workflow Performance Observations

Agent Container Smoke Test Results

Run §23563730026 (success): All 12/12 tools verified:
bash, sh, git, jq, yq, curl, gh, node, python3, go, java, dotnet

Run §23563803877: Also 12/12 ✅ — note java version reported as 10.0.102 (dotnet version) suggesting a display artifact.

Agentic Assessment: Daily Workflow Updater

Assessment: overkill_for_agentic (severity: low)

"This Repo Maintenance run looks stable enough that deterministic automation may be a simpler fit."

The workflow's task (checking for action version updates and creating a PR) is highly deterministic and could potentially be replaced by a scripted workflow step. However, the agent completed the task successfully and the assessment is low severity.

---

Recommendations

1. Fix Copilot CLI auth in detection job — The Daily Workflow Updater fails its detection phase due to auth issues. This masks whether the agent output was safe.
2. Scope Smoke Copilot more carefully — 12 write actions + high friction rating suggests the smoke test does too much in production. Consider a lighter-touch smoke test mode.
3. Register Serena MCP or remove from smoke test — Either provision Serena or update the smoke test to not flag it as a failure if it's intentionally not available in this environment.
4. API Rate Limiting — The GitHub API rate limit was hit during log fetching. Consider staggering audit runs or using cached data more aggressively.

---

References:

• §23563694023 — Daily Workflow Updater (failure)
• §23563669474 — Smoke Copilot (partial pass)
• §23563889311 — This audit run

AI generated by Agentic Workflow Audit Agent · history

• expires on Mar 26, 2026, 9:30 PM UTC

[github-actions[bot]]

💥 WHOOSH!

ZAP! The Claude smoke test agent was HERE, citizen! 🦸

POW! Run §23565361241 swooped in, tested all the things, and emerged VICTORIOUS!

BOOM! All systems: NOMINAL! 🚀

— Claude Engine, signing off ⚡

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Agentic Workflow Audit Agent.

A newer discussion is available at Discussion #23173.