Safe Output Health Report — 2026-03-26

[github-actions[bot]]

Executive Summary

• Period: 2026-03-26 (last 24 hours)
• Runs Analyzed: 180 workflow runs downloaded
• Runs with Accessible Safe Output Data: 22
• Agent-Produced Safe Output Items: 32 items across 12 workflows
• Safe Output Items Successfully Executed: 28 items (100% execution success rate)
• Safe Output Jobs Failed: 0 — all safe output jobs that ran completed successfully
• Items Blocked by Upstream Failures: 8 items across 4 runs (blocked by detection gate)
• Error Clusters Identified: 2

The safe output job infrastructure is healthy. No safe output job failures occurred today. The primary concerns are upstream issues — detection model format failures that gate the safe_outputs job, and a push_repo_memory job infrastructure error.

---

Safe Output Job Statistics

Job Type	Executions	Failures	Success Rate
create_discussion	4	0	100%
create_issue	4	0	100%
add_comment	6	0	100%
create_pull_request_review_comment	2	0	100%
submit_pull_request_review	2	0	100%
update_pull_request	1	0	100%
add_reviewer	1	0	100%
send_slack_message / post_slack_message	2	0	100%
dispatch_workflow	1	0	100%
set_issue_type	1	0	100%
assign_to_agent	3	0	100%
Total	27	0	100%

Note: assign_to_agent items run as standalone steps and are tracked separately from the main safe-output-items artifact. All 3 Copilot assignments succeeded for Issue Monster.

---

Upstream Issues Blocking Safe Outputs

Issue 1: Detection Model Format Failure (4 runs affected)

The most impactful issue is the Copilot detection model failing to produce the required THREAT_DETECTION_RESULT JSON format, which causes the detection job to fail with:

❌ Failed to parse detection result: No THREAT_DETECTION_RESULT found in detection log.
   The detection model may have failed to follow the output format.
```

Because the `safe_outputs` job requires `needs.detection.result == 'success'`, detection failures prevent **valid agent outputs from being executed**. Affected runs and lost items:

| Run | Workflow | Lost Items |
|-----|----------|-----------|
| [§23593365566](https://github.com/github/gh-aw/actions/runs/23593365566) | Delight | `create_discussion` (UX Analysis Report) |
| [§23594917482](https://github.com/github/gh-aw/actions/runs/23594917482) | Auto-Triage Issues | `create_discussion` + 3× `add_labels` |
| [§23593121534](https://github.com/github/gh-aw/actions/runs/23593121534) | Dead Code Removal Agent | `create_pull_request` |
| [§23594893714](https://github.com/github/gh-aw/actions/runs/23594893714) | Smoke Gemini | `create_issue` |

**Root Cause**: The Copilot model running the threat detection task failed to emit the expected format line. The 120-line detection logs were produced but lacked the required `THREAT_DETECTION_RESULT:{...}` structured output.

**Impact**: HIGH — these are legitimate agent outputs that should have been published. This is a new pattern not seen in prior audits (first observed today).

#### Issue 2: `push_repo_memory` Job Missing `actions/setup` (2 runs affected)

Two workflows with a `push_repo_memory` job failed with:

```
Can't find 'action.yml', 'action.yaml' or 'Dockerfile' under 
'/home/runner/work/gh-aw/gh-aw/actions/setup'. Did you forget to run 
actions/checkout before running your local action?

Affected runs:

• §23593331592 — Workflow Health Manager (safe_outputs ran and succeeded with 2 items, but overall workflow = failure)
• §23593690977 — PR Triage Agent (safe_outputs ran, noop posted to issue [aw] No-Op Runs #21483, but workflow = failure)

Root Cause: The push_repo_memory job cannot find the local actions/setup action, suggesting a checkout ordering issue — the actions folder checkout likely needs to precede the local action reference.

Impact: MEDIUM — safe_outputs job succeeds and executes correctly, but the overall workflow conclusion is marked as failure due to this upstream job failure.

---

Successful Run Highlights

View successful workflow executions

Run	Workflow	Engine	Items Executed
§23595269467	Smoke Copilot	copilot	8 (add_comment, create_issue, create_discussion, etc.)
§23594519542	Smoke Claude	claude	8 (full smoke test suite)
§23594395837	Issue Monster	copilot	6 (3× add_comment + 3× assign_to_agent)
§23593024278	Duplicate Code Detector	codex	1 (create_issue)
§23593035940	Terminal Stylist	copilot	1 (create_discussion)
§23593331592	Workflow Health Manager	copilot	2 (add_comment, create_issue)
§23593355602	Multi-Device Docs Tester	claude	1 (create_issue)
§23595255460	Constraint Solving	copilot	1 (create_discussion)

---

Recommendations

Critical (Investigate Detection Failures)

[EP024] Copilot Detection Model THREAT_DETECTION_RESULT Format Failure — NEW

• Priority: High
• Root Cause: Copilot model runs the detection prompt but does not produce the required structured output line THREAT_DETECTION_RESULT:{...} in at least 4 runs today
• Affected Workflows: Delight, Auto-Triage Issues, Dead Code Removal Agent, Smoke Gemini
• Recommended Action:
  1. Review the detection prompt template to ensure Copilot follows output format instructions
  2. Check if a recent model update or prompt change affected detection reliability
  3. Consider adding retry logic (1-2 retries) when THREAT_DETECTION_RESULT is not found
  4. Monitor tomorrow's audit for recurrence

Medium Priority (Infrastructure)

[EP025] push_repo_memory Missing actions/setup — NEW

• Priority: Medium
• Root Cause: The actions/setup local action cannot be found during push_repo_memory job execution; likely checkout order issue
• Affected Workflows: Workflow Health Manager, PR Triage Agent
• Recommended Action: Verify that the Checkout actions folder step runs before any local action references in the push_repo_memory job YAML. The error suggests the checkout either didn't run or ran after the local action reference.

---

Work Items

Work Item 1: Investigate and Fix Detection Model Format Failures

• Type: Investigation / Bug Fix
• Priority: High
• Description: Copilot detection model fails to produce THREAT_DETECTION_RESULT in ~15-20% of detection runs today, causing valid agent outputs to be silently dropped. 4 workflows lost their output today.
• Acceptance Criteria:
  • Root cause identified (prompt change? model version? edge case in agent output size?)
  • Detection format failure rate drops to near 0%
  • Consider adding retry or fallback for detection failures
• Technical Approach: Review detection prompt, compare detection log content for failing vs passing runs, check if output token limits are being hit
• Estimated Effort: Medium

Work Item 2: Fix push_repo_memory Checkout Order

• Type: Bug Fix
• Priority: Medium
• Description: push_repo_memory job fails because actions/setup is not found, marking runs as failed even when all safe_outputs succeeded
• Acceptance Criteria:
  • push_repo_memory job succeeds in Workflow Health Manager and PR Triage Agent runs
  • Overall workflow conclusion reflects actual safe output health
• Technical Approach: Review compiled YAML for workflows using push_repo_memory; ensure Checkout actions folder step precedes local action usage
• Estimated Effort: Small

---

Historical Context

This is audit run #188 for the Safe Output Health Monitor. Prior audits show:

• 2026-03-25: 100% safe output success rate, 0 failures, 7 items executed across 3 workflows
• 2026-03-26 (today): 100% safe output success rate, 28 items executed across 12 workflows; 2 new upstream patterns identified

The safe output job infrastructure has maintained a 100% execution success rate for all items that reached the safe_outputs job. The recurring patterns (EP002, EP008) from earlier audits were not observed in today's 24-hour window.

Trends:

• Safe output job health: ✅ Stable (100% success for items that run)
• Detection model reliability: ⚠️ New concern — 4 format failures today
• Push repo-memory: ⚠️ New infrastructure issue affecting 2 workflows

---

Next Steps

• Investigate EP024: Check detection prompt and Copilot model behavior for today's failing runs
• Fix EP025: Correct push_repo_memory job checkout order
• Monitor tomorrow's audit for EP024 recurrence
• Consider adding a detection failure rate metric to the daily KPIs

References:

• §23593365566 (Delight — detection failure)
• §23594917482 (Auto-Triage Issues — detection failure)
• §23595269467 (Smoke Copilot — 8 items success)

AI generated by Safe Output Health Monitor · history

• expires on Mar 27, 2026, 1:24 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #23237.