[daily regulatory] Regulatory Report - 2026-03-31

[github-actions[bot]]

Today's analysis reviewed 21 daily report discussions generated in the last 48 hours for the github/gh-aw repository. Overall data quality is good — reports are consistent, methodologically sound, and present complementary (rather than conflicting) views of the same system. No critical cross-report contradictions were identified. Three medium-severity findings were detected: a known safe-output infrastructure failure (missing cross-repo test branch), a documented efficiency concern (Daily Syntax Error Quality Check at 90 turns/run), and declining scores in the Agent Performance Report (-3 quality, -2 effectiveness). Three P1 issues remain open from prior days.

The standout positive signal: Copilot agent PR merge rate reached 90.3% today — the highest in the tracked 3-day window (83% → 87% → 90%) — while average PR duration dropped dramatically from 161 min to 49 min. Token consumption dropped 51.4% vs. prior period, driven by fewer runs rather than efficiency gains.

📋 Full Regulatory Report

📊 Reports Reviewed

#	Report	Created	Status
#23677	[copilot-agent-analysis] Daily Copilot Agent Analysis - 2026-03-31	11:41 UTC	✅ Valid
#23678	Daily Copilot Token Consumption Report — 2026-03-31	11:44 UTC	✅ Valid
#23680	[copilot-session-insights] Daily Copilot Agent Session Analysis — 2026-03-31	11:50 UTC	✅ Valid
#23681	Daily Firewall Report - 2026-03-31	11:58 UTC	✅ Valid
#23686	UX Analysis Report — 2026-03-31	12:11 UTC	✅ Valid
#23704	[Auto-Triage] Auto-Triage Report — 2026-03-31	12:49 UTC	✅ Valid
#23723	Safe Output Health Report - 2026-03-31	13:21 UTC	⚠️ 1 Failure
#23730	🔬 Repository Quality Improvement Report	13:39 UTC	✅ Valid
#23642	Agent Performance Report — Week of 2026-03-31	04:47 UTC	⚠️ Score decline
#23745	[copilot-pr-merged-report] Daily Copilot PR Merged Report - 2026-03-31	15:35 UTC	✅ Valid
#23748	📰 Repository Chronicle	16:29 UTC	✅ Valid
#23761	[Auto-Triage] Auto-Triage Report — 2026-03-31	18:53 UTC	✅ Valid
#23763	Static Analysis Report - 2026-03-31	19:52 UTC	✅ Valid
#23770	[sergo] Sergo Report: Mutex Safety & Compile-Time Interface Checks	20:32 UTC	✅ Valid
#23772	[integrity] DIFC Integrity-Filtered Events Report — 2026-03-31	20:37 UTC	✅ Valid
#23775	[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-03-31	20:44 UTC	✅ Valid
#23603	[daily performance] Daily Performance Summary - 2026-03-30	23:30 UTC (Mar 30)	✅ Valid
#23595	Daily Code Metrics Report - 2026-03-30	21:57 UTC (Mar 30)	✅ Valid
#23527	[observability] Agentic Observability Report — Mar 16–30, 2026	08:54 UTC (Mar 30)	✅ Valid
#23526	🏥 Organization Health Report — github org	08:53 UTC (Mar 30)	✅ Valid
#23596	[daily secrets] Daily Secrets Analysis — 2026-03-30	22:23 UTC (Mar 30)	✅ Valid

🔍 Data Consistency Analysis

Cross-Report Metrics Comparison

Metric definitions and scopes reference scratchpad/metrics-glossary.md.

Metric	Copilot Agent Analysis	PR Merged Report	Session Insights	Scope	Status
Agent PRs merged (24h)	28 (90.3%)	41 total merged	—	⚠️ Different scopes	ℹ️ See Note A
Session completion rate	—	—	46.9% (15/32)	⚠️ Different scope	ℹ️ See Note B
Total runs (token cycle)	—	—	—	—	✅ Consistent
Firewall-enabled runs	19 (Firewall)	—	105 (Obs, 14d)	⚠️ Different windows	ℹ️ See Note C

Scope Notes:

• Note A: Agent PR count (28 copilot-authored) is a subset of all merged PRs (41). The 13-PR difference represents human-reviewed or non-copilot PRs. This is expected behavior, not a discrepancy.
• Note B: Session completion (46.9%) measures whether a Copilot session finishes successfully vs. PR merge rate (90.3%), which measures whether agent-created PRs get merged by humans. These are fundamentally different metrics covering different phases of the agent lifecycle.
• Note C: Firewall report (19 runs, single day) vs. Observability report (105 runs, 14 days) — different time windows. Daily average from observability is ~7.5 runs/day, consistent with 19 runs on a higher-activity day.

Internal Math Checks

Report	Check	Result
Firewall: 577 allowed + 1 blocked = 578 total	✅ Verified	Pass
Safe Output: 6 success + 1 fail = 7 messages	✅ Verified	Pass
Copilot PRs: 28 merged + 3 not-merged = 31 total	✅ Verified	Pass
Token: 35.7M (top 10) = 44.1% of 80.9M	✅ ~44.1% matches	Pass
Prompt Clustering: 117+57+52+190+43+66+78+63+24 = 690 merged of 1000	✅ 69% confirmed	Pass

Consistency Score

• Overall Consistency: 92% (metrics internally consistent within each report)
• Cross-Report Alignment: All differences explained by intentional scope differences
• Critical Discrepancies: 0
• Minor Scope Differences (documented): 3

⚠️ Issues and Anomalies

Medium-Severity Issues

1. Safe Output: Missing Cross-Repo Branch

   • Affected Report: Safe Output Health Report #23723
   • Metric: push_to_pull_request_branch success rate = 0%
   • Description: Smoke Update Cross-Repo PR workflow failed to find branch pr-branch in githubnext/gh-aw-side-repo
   • Expected: Branch should exist as smoke test fixture
   • Actual: Fatal git error — branch not found; likely merged/deleted
   • Severity: Medium — graceful degradation worked, but smoke test infrastructure is broken
   • Recommended Action: Investigate githubnext/gh-aw-side-repo PR rejig docs #1 / branch pr-branch status; recreate if deleted

2. Token Efficiency: Runaway Turn Count

   • Affected Report: Token Consumption Report #23678
   • Metric: Avg turns/run = 14.7; max = 90 turns (Daily Syntax Error Quality Check)
   • Description: Daily Syntax Error Quality Check used 90 turns in a single run (4.5M tokens)
   • Expected: Complex analysis workflows typically 15–40 turns
   • Actual: 90 turns — 2–6× the norm; likely excessive exploration or inadequate early-exit conditions
   • Severity: Medium — cost impact, but run completed
   • Recommended Action: Review prompt for Daily Syntax Error Quality Check to add deterministic pre-steps and tighter scope constraints

3. Agent Performance Score Decline

   • Affected Report: Agent Performance Report #23642
   • Metric: Quality score 79/100 (↓3 from 82), Effectiveness 73/100 (↓2 from 75)
   • Description: Week-over-week decline in both quality and effectiveness scores
   • Severity: Medium — trend to watch; 3 P1 issues remain open
   • Recommended Action: Prioritize P1 resolution (Smoke Codex OpenAI API, Cross-Repo PR smoke tests)

Informational Notes

• Smoke Codex (P1): ALL run types failing since 2026-03-28 due to OpenAI API access restriction. Issue [aw] Smoke Codex failed #23431.
• Smoke Cross-Repo PRs (P1): Failures from push_repo_memory → post-setup-scripts bug. Issues [aw] Smoke Update Cross-Repo PR failed #23193/[WHM] Smoke Create Cross-Repo PR - Persistent Schedule Failures (Systemic Bug) #23447. Root cause identified, fix pending.
• Documentation Unbloat: 100% failure rate this window (single run, fast failure). Issue [agent-perf] Documentation Unbloat: 100% failure rate — needs investigation #23640 created.
• Smoke Claude: PR-triggered runs consistently resource-heavy (39 turns, 1.51M tokens). Issues [aw] Smoke Claude failed #23067/[observability escalation] Smoke test workflows repeatedly exceed resource and control thresholds (Smoke Claude, Smoke Copilot) #23528.
• Token decrease (-51.4%): Not an anomaly — explained by fewer runs in current cycle (111 vs prior period), not efficiency improvement.

📈 Trend Analysis

Day-over-Day (Mar 29 → Mar 30 → Mar 31)

Metric	Mar 29	Mar 30	Mar 31	Trend
Agent PR merge rate	83.3%	87.0%	90.3%	📈 Improving
Avg PR duration	161 min	130 min	49 min	📈 Dramatically faster
Safe output failures	4	5	1	📈 Improving
Copilot sessions	19	19	32	📈 Increasing volume
Session completion rate	~42.1%	~42.1%	46.9%	📈 Improving

Notable Trends

• Agent quality is accelerating: Merge rate up 7pp and duration cut by 70% in 3 days — strongest performance window this month
• Safe output stability improving: 1 failure today vs 4-5 in prior days; new failure pattern (push_to_pull_request_branch) is infrastructure-specific, not systemic
• Copilot session volume surging: 32 sessions Mar 31 vs 5-19 in prior days — rapid adoption signal
• Code quality metric divergence: Code Metrics shows 73/100 quality while Agent Performance shows 79/100 — these measure different things (source code quality vs agent workflow effectiveness)

📝 Per-Report Analysis

Copilot Agent Analysis & Session Insights

Sources: #23677 · #23680

Metric	Value
Agent PRs (24h)	31
Merged	28 (90.3%)
Avg duration	49 min
Total sessions	50
Copilot sessions	32
Session completions	15 (46.9%)

Notes: Session completion (46.9%) vs merge rate (90.3%) divergence is expected — many sessions belong to review agents that return action_required by design, not genuine failures.

Token Consumption Report

Source: #23678

Metric	Value
Total tokens	80,923,350
Total runs	111 (66 unique workflows)
Total turns	1,627
Action minutes	890 (14.8 hrs)
Avg tokens/run	729,039
Avg turns/run	14.7
Change vs prior	-51.4% tokens, -43.1% runs

Notes: Decrease is volume-driven (fewer runs), not efficiency improvement. High per-run averages (729k tokens) warrant efficiency review for top consumers.

Daily Firewall Report

Source: #23681

Metric	Value
Firewall-enabled runs	19
Runs with audit logs	15
Total requests	578
Allowed	577 (99.8%)
Blocked	1 (0.17%)
Blocked domains	1 (docs.astro.build)

Notes: Excellent firewall health. Single blocked domain is a legitimate documentation site — recommend adding to Update Astro workflow allowlist.

Safe Output Health Report

Source: #23723

Metric	Value
Total runs analyzed	67
Safe output messages	7
Successes	6 (85.7%)
Failures	1 (14.3%)
Error clusters	1

Notes: The two recurring error patterns from prior days were not observed today — a positive improvement. New failure is infrastructure-specific (missing branch in side-repo).

Prompt Clustering Analysis

Source: #23775

Metric	Value
PRs analyzed	1,000 (2026-01-21 → 2026-03-31)
Merged	690 (69%)
Clusters	9
Best cluster	Cluster H/I: 80% success (Safe Outputs / CI Job URLs)
Worst cluster	Cluster A: 53% success (CI Failure Issue-Driven Repairs)

Notes: Historical analysis. The 27pp gap between structured (CI Job URL) and unstructured (issue report) prompts is the key actionable finding.

Code Metrics & Performance Summary

Sources: #23595 · #23603

Metric	Value
Total LOC	719,100
Quality score	73/100
Test-to-source ratio	2.189
PRs merged (90d)	84
Avg merge time	1.9 hours
Issues resolved (90d)	891
Issue resolution rate	89.1%

💡 Recommendations

Process Improvements

1. Restore cross-repo smoke test fixture: The pr-branch in githubnext/gh-aw-side-repo appears to be missing. Recreate PR rejig docs #1 and branch to re-enable the Smoke Update/Create Cross-Repo PR smoke tests. This also unblocks P1 issues [aw] Smoke Update Cross-Repo PR failed #23193 and [WHM] Smoke Create Cross-Repo PR - Persistent Schedule Failures (Systemic Bug) #23447.

2. Add turn-count guardrails to high-consumption workflows: Daily Syntax Error Quality Check (90 turns) and Code Simplifier (66 turns) exceed typical bounds. Add max_turns limits and deterministic pre-steps to reduce exploration depth.

3. Adopt structured prompt templates more broadly: Prompt Clustering analysis shows structured CI Job URL prompts achieve 80% merge rate vs 53% for unstructured issue-driven prompts. Apply this pattern to the CI Failure Doctor and deep-report auto-generated prompts.

4. Add docs.astro.build to Update Astro allowlist: The single blocked domain is legitimate; a trivial one-line fix eliminates the false positive.

Data Quality Actions

1. Standardize session success definition: The gap between session completion rate (46.9%) and PR merge rate (90.3%) is expected but confusing to stakeholders. Consider adding a note to the Session Insights report clarifying that action_required from review agents is a success, not a failure.

2. Track token efficiency trends: The current token report tracks total consumption but not efficiency-over-time. Adding a "tokens per successfully merged PR" metric would better capture productivity.

📊 Regulatory Summary

Metric	Value
Reports Reviewed	21
Reports Passed	19
Reports with Issues (Medium)	2
Reports Failed	0
Critical Discrepancies	0
Math Validation Checks Passed	5/5
P1 Issues Open	3
Overall Health Score	88%

---

References:

• §23819442273 — This regulatory run
• §23816894640 — Prompt Clustering Analysis source
• §23735933506 — Org Health Report source

AI generated by Daily Regulatory Report Generator · history

• expires on Apr 3, 2026, 9:17 PM UTC

[github-actions[bot]]

🤖 Smoke test agent was here! 👋 Running diagnostics across the galaxy... all systems nominal. The robots are watching — and approving of your regulatory compliance. 🚀✨

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

This discussion has been marked as outdated by Daily Regulatory Report Generator.

A newer discussion is available at Discussion #23953.

[github-actions[bot]]

This report has been superseded by a newer daily regulatory report for 2026-04-01.