🏥 Safe Output Health Report - November 24, 2025

[github-actions[bot]]

🏥 Safe Output Health Report - November 24, 2025

This report provides an audit of safe output job health over the last 24 hours in the githubnext/gh-aw repository.

Executive Summary

Over the last 24 hours, 124 workflow runs generated 263 safe output jobs. The analysis reveals two distinct health concerns:

1. High Skip Rate (PRIMARY CONCERN): 72.6% (191/263) of safe output jobs are being skipped
2. Permission Errors (SECONDARY): All 8 actual failures (100%) are due to token permissions, not code defects

Key Insight: The safe output infrastructure is working correctly when invoked - the real issue is that agents aren't generating output for these job types as frequently as expected.

Safe Output Job Statistics

Metric	Count	Percentage
Total Runs Examined	124	-
Total Safe Output Jobs	263	100%
Successful Executions	64	24.3%
Failed Executions	8	3.0%
Skipped Jobs	191	72.6%
Execution Rate	72	27.4%

Job Type Breakdown

Job Type	Total	Success	Failure	Skipped	Success Rate*	Execution Rate
create_discussion	21	17	0	4	81.0%	81.0%
add_comment	44	18	0	26	40.9%	40.9%
push_to_pull_request_branch	19	6	0	13	31.6%	31.6%
create_pull_request	35	8	6	21	22.9%	40.0%
create_issue	22	3	2	17	13.6%	22.7%
missing_tool	122	12	0	110	9.8%	9.8%

*Success rate = successful / total jobs

Full Report Details

Detailed Analysis

Issue 1: High Safe Output Skip Rate (PRIMARY CONCERN)

Impact: 191 out of 263 (72.6%) safe output jobs were skipped

Root Cause: Safe output jobs are skipped when agents don't generate output for that specific safe output type. This indicates:

1. Workflow design: Some workflows may not be designed to produce certain safe output types
2. Agent behavior: Agents are not consistently using safe output tools when appropriate
3. Conditional execution: Jobs are correctly skipped when no relevant agent output exists

Breakdown by Job Type:

Job Type	Skipped Count	Skip Rate
missing_tool	110	90.2%
add_comment	26	59.1%
create_pull_request	21	60.0%
create_issue	17	77.3%
push_to_pull_request_branch	13	68.4%
create_discussion	4	19.0%

Analysis:

• missing_tool (90.2% skip rate): This is expected - missing_tool reports should only be generated when tools are unavailable. High skip rate indicates good tool availability.
• create_discussion (19.0% skip rate): Best execution rate, indicating workflows are designed to produce discussion output
• create_issue (77.3% skip rate): Low execution rate suggests workflows aren't generating issue creation requests
• create_pull_request (60.0% skip rate): Moderate execution rate for PR creation workflows

Issue 2: Permission-Related Failures (SECONDARY)

Impact: 8 failures across 2 job types (100% of all failures are permissions-related)

Root Cause: Personal Access Token (PAT) used by workflows lacks specific permissions for:

1. Assigning PR reviewers (6 failures)
2. Assigning issues to users (2 failures)

Error Cluster 1: PR Reviewer Assignment Failures

• Count: 6 failures
• Affected Job: create_pull_request
• Pattern: All failures occur when attempting to assign copilot-pull-request-reviewer[bot] as a reviewer
• Error Message: "Resource not accessible by personal access token"
• API Call: POST /repos/{owner}/{repo}/pulls/{pull_number}/requested_reviewers

Affected Runs:

• §19544887449
• §19587010229
• §19590023376
• §19606921189
• §19612464420
• §19616221625

Technical Details:

gh api --method POST /repos/githubnext/gh-aw/pulls/$PR_NUMBER/requested_reviewers \
  -f 'reviewers[]=copilot-pull-request-reviewer[bot]'

Error: gh: Resource not accessible by personal access token (HTTP 403)

Error Cluster 2: Issue Assignment Failures

• Count: 2 failures
• Affected Job: create_issue
• Pattern: Failures when assigning issues to @copilot
• Error Message: "GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)"
• API Call: gh issue edit {issue_number} --add-assignee @copilot``

Affected Runs:

• §19601153269
• §19617204957

Technical Details:

gh issue edit 4572 --add-assignee `@copilot`

Error: failed to update https://github.com/githubnext/gh-aw/issues/4572:
GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)

Root Cause Analysis

API-Related Issues

No API rate limiting, timeouts, or connectivity issues detected. All API failures are permission-related.

Permission Issues

Finding: PAT tokens used for safe output operations lack necessary scopes for:

1. PR Reviewer Requests: Requires enhanced permissions beyond basic PR write access
2. Issue Assignment: GraphQL mutation replaceActorsForAssignable requires specific permissions

Note: The primary safe output operations (creating issues, PRs, comments, discussions) work correctly. Only post-creation assignment operations fail.

High Skip Rate Analysis

Finding: The 72.6% skip rate is not a failure - it's an indicator of workflow behavior:

1. By Design: Not all workflows are designed to produce all safe output types
2. Conditional Logic: Safe output jobs correctly skip when agents don't generate relevant output
3. Agent Discretion: Agents make contextual decisions about what output types to generate

Positive Indicators:

• When jobs execute, success rates are strong (81% for discussions, 41% for comments)
• Skipped jobs don't consume resources or cause errors
• The infrastructure correctly handles conditional execution

Recommendations

Critical Issues (Immediate Action Required)

1. Token Permissions for Post-Creation Operations

• Priority: High
• Root Cause: PAT lacks permissions for reviewer requests and issue assignments
• Recommended Action: Document that these post-creation operations require enhanced token permissions or consider making them optional
• Affected: create_pull_request (6 failures), create_issue (2 failures)
• Impact: Medium - core functionality (PR/issue creation) works, but automation features fail

Process Improvements

1. Graceful Degradation for Permission Errors

• Current State: Jobs fail when post-creation operations (reviewer assignment, issue assignment) encounter permission errors
• Proposed: Catch permission errors and downgrade to warnings, allowing the job to succeed
• Benefits:
  • Jobs marked as successful when primary objective (create PR/issue) completes
  • Clear logging indicates which optional features require elevated permissions
  • Reduces noise in failure metrics

2. Safe Output Usage Guidelines

• Current State: High skip rates indicate unclear guidance on when to use each safe output type
• Proposed: Document recommended usage patterns for each safe output type
• Benefits:
  • Workflow authors understand when to expect each job type
  • Agents receive clearer instructions on output generation
  • Skip rates align with expectations

3. Skip Rate Monitoring

• Current State: Skip rates are tracked but not contextualized
• Proposed: Establish baseline skip rates per job type and workflow
• Benefits:
  • Distinguish expected skips from concerning trends
  • Identify workflows with unexpectedly low execution rates
  • Set realistic success rate targets

Work Item Plans

Work Item 1: Make Post-Creation Operations Optional

• Type: Enhancement
• Priority: High
• Description: Modify create_pull_request and create_issue jobs to gracefully handle permission errors in post-creation steps (reviewer assignment, issue assignment)
• Acceptance Criteria:
  • PR creation succeeds even when reviewer assignment fails
  • Issue creation succeeds even when assignee update fails
  • Permission errors are logged as warnings, not failures
  • Job summaries clearly indicate which operations succeeded vs. required elevated permissions
• Technical Approach:
  1. Wrap reviewer request API call in try-catch with specific permission error handling
  2. Wrap issue assignment API call in try-catch with GraphQL permission error handling
  3. Update job conclusion logic to consider primary operation (create) vs. secondary (assign)
  4. Add detailed logging explaining permission requirements
• Estimated Effort: Small
• Dependencies: None
• Files to Modify:
  • .github/workflows/*.lock.yml (create_pull_request jobs)
  • .github/workflows/*.lock.yml (create_issue jobs)

Work Item 2: Document Token Permission Requirements

• Type: Documentation
• Priority: High
• Description: Create comprehensive documentation of token permissions required for each safe output operation
• Acceptance Criteria:
  • Document minimum permissions for each safe output job type
  • Document optional enhanced permissions and their benefits
  • Provide example token configurations
  • Include troubleshooting guide for permission errors
• Technical Approach:
  1. Test each safe output type with minimal permissions
  2. Identify which operations require elevated permissions
  3. Create permission matrix table
  4. Add to safe outputs documentation
• Estimated Effort: Small
• Dependencies: None

Work Item 3: Establish Skip Rate Baselines

• Type: Investigation
• Priority: Medium
• Description: Analyze historical skip rates to establish baselines for each job type and workflow
• Acceptance Criteria:
  • Calculate 30-day average skip rates per job type
  • Calculate skip rates per workflow
  • Identify outlier workflows with unusually high/low skip rates
  • Document expected vs. concerning skip rate ranges
• Technical Approach:
  1. Query last 30 days of workflow run logs
  2. Aggregate skip rates by job type and workflow
  3. Calculate percentiles (25th, 50th, 75th, 95th)
  4. Create baseline reference document
• Estimated Effort: Medium
• Dependencies: Historical log data availability

Work Item 4: Safe Output Usage Guidelines

• Type: Documentation
• Priority: Low
• Description: Create guidelines for workflow authors on when to use each safe output type
• Acceptance Criteria:
  • Document use cases for each safe output type
  • Provide decision tree for choosing output types
  • Include example workflows demonstrating each type
  • Add anti-patterns (when NOT to use each type)
• Technical Approach:
  1. Review existing workflows using each safe output type
  2. Identify common patterns and use cases
  3. Create workflow templates
  4. Add to workflow authoring guide
• Estimated Effort: Medium
• Dependencies: Work Item 2 (permission documentation)

Historical Context

This is the first audit of safe output job health using the new monitoring infrastructure. No historical trend data is available yet. Future audits will include:

• Comparison with previous audit periods
• Trend analysis (improving/degrading/stable)
• Recurring vs. new issues
• Resolution tracking for identified issues

Metrics and KPIs

Overall Health Metrics

• Overall Safe Output Success Rate: 88.9% (64 successes / 72 executed jobs)
• Overall Execution Rate: 27.4% (72 executed / 263 total jobs)
• Failure Rate of Executed Jobs: 11.1% (8 failures / 72 executed jobs)
• Most Reliable Job Type: create_discussion (81.0% success rate)
• Least Reliable Job Type: create_issue (13.6% success rate, but 100% of failures are permission-related)

Health Indicators

Indicator	Status	Notes
Code Quality	🟢 Healthy	No code bugs detected; all failures are permission-related
API Integration	🟢 Healthy	No rate limits, timeouts, or connectivity issues
Permission Configuration	🟡 Attention Needed	8 failures due to insufficient PAT permissions
Execution Rate	🟡 Attention Needed	72.6% skip rate requires baseline establishment
Error Handling	🟢 Healthy	Jobs gracefully skip when no output generated

Next Steps

• Complete initial safe output health audit
• Implement graceful degradation for permission errors (Work Item 1)
• Document token permission requirements (Work Item 2)
• Establish skip rate baselines (Work Item 3)
• Create safe output usage guidelines (Work Item 4)
• Schedule follow-up audit in 7 days to measure improvements

---

References:

• §19544887449
• §19601153269
• §19606921189

AI generated by Safe Output Health Monitor

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.