🏥 Safe Output Health Report - 2025-11-23

[github-actions[bot]]

🏥 Safe Output Health Report - 2025-11-23

Executive Summary

Daily health audit of safe output jobs reveals improving reliability with targeted issues requiring attention. The overall safe output success rate has improved from 88.9% to 90.6%, with failure count decreasing from 12 to 5.

Period: Last 24 hours (November 22-23, 2025)
Runs Analyzed: 101 workflow runs
Safe Output Jobs Executed: 210 jobs
Safe Output Jobs Failed: 5 jobs
Success Rate: 90.6% (48 successes out of 53 non-skipped jobs)
Error Clusters Identified: 2 primary clusters

Safe Output Job Statistics

Safe Output Job Execution Summary

Job Type	Total Executions	Successes	Failures	Skipped	Cancelled	Success Rate
missing_tool	97	9	0	84	4	100%
add_comment	32	21	0	11	0	100%
create_issue	26	4	3	19	0	57%
create_pull_request	25	5	1	15	4	83%
push_to_pull_request_branch	19	6	1	8	4	86%
create_discussion	11	8	0	3	0	100%
TOTAL	210	48	5	156	12	90.6%

Key Observations

• Most Reliable Jobs: add_comment (100%), create_discussion (100%), missing_tool (100%)
• Most Problematic Job: create_issue (57% success rate, 3 failures)
• High Skip Rate: 156 jobs skipped (74.3%) - expected when no output generated

Error Clusters

Cluster 1: GitHub API Permission Errors (4 failures)

Severity: High
Pattern: Token lacks required permissions for secondary operations after primary operation succeeds
Root Cause: GITHUB_TOKEN or GH_TOKEN used in safe output jobs lacks specific GraphQL and REST API permissions

Affected Runs

1. §19516274301 - Duplicate Code Detector

   • Job: create_issue
   • Primary Operation: ✅ Created issue #4362
   • Secondary Operation: ❌ Failed to assign to @copilot
   • Error: GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)

2. §19551191192 - Daily File Diet

   • Job: create_issue
   • Primary Operation: ✅ Created issue #4436
   • Secondary Operation: ❌ Failed to assign to @copilot
   • Error: GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)

3. §19601153269 - Duplicate Code Detector

   • Job: create_issue
   • Primary Operation: ✅ Created issue #4572
   • Secondary Operation: ❌ Failed to assign to @copilot
   • Error: GraphQL: Resource not accessible by personal access token (replaceActorsForAssignable)

4. §19590718399 - Security Fix PR

   • Job: create_pull_request
   • Primary Operation: ✅ Created PR #4540
   • Secondary Operation: ❌ Failed to request reviewers
   • Error: Resource not accessible by personal access token (HTTP 403) when calling /repos/{owner}/{repo}/pulls/{pr}/requested_reviewers

Technical Analysis

The safe output jobs execute a two-phase operation:

Phase 1 (Primary): Create the resource (issue/PR) - SUCCEEDS
Phase 2 (Secondary): Perform additional operations (assign, request reviewers) - FAILS

The current token has permissions for:

• ✅ issues: write - can create issues
• ✅ pull_requests: write - can create PRs

But lacks permissions for:

• ❌ GraphQL replaceActorsForAssignable mutation
• ❌ REST API /pulls/{pr}/requested_reviewers endpoint

Cluster 2: GitHub App Configuration Error (1 failure)

Severity: Critical
Pattern: Invalid or malformed GitHub App private key preventing authentication
Root Cause: The GH_APP_PRIVATE_KEY secret contains invalid key data

Affected Run

1. §19586088247 - Unbloat Docs
   • Job: push_to_pull_request_branch
   • Operation: ❌ Failed to create GitHub App token
   • Error: Invalid keyData → Failed to read private key → DOMException [DataError]
   • Impact: Unable to authenticate as GitHub App, completely blocking the push operation

Technical Analysis

The push_to_pull_request_branch job requires a GitHub App token with elevated permissions to push changes. The job attempts to:

1. Load the GitHub App private key from the GH_APP_PRIVATE_KEY secret
2. Create a JWT for GitHub App authentication
3. Exchange JWT for an installation access token

Failure Point: Step 1 - the private key fails to parse as valid PEM-encoded RSA key data

Possible causes:

• Key was not properly copied/pasted (missing headers/footers)
• Key contains invalid characters or formatting
• Key is corrupted or truncated
• Wrong key type (e.g., public key instead of private key)

Root Cause Analysis

Permission Issues (Cluster 1)

Primary Issue: Separation of concerns in GitHub token permissions

GitHub's permission model distinguishes between:

1. Basic write operations (create issue, create PR) - available to standard tokens
2. Advanced operations (assign, request reviewers) - require additional scopes or app permissions

The current implementation treats these as a single atomic operation, causing the entire job to fail when secondary operations are denied, even though the primary operation succeeded.

Impact Assessment:

• Functional Impact: Medium - Issues and PRs are created successfully, but missing assignments/reviewers
• Operational Impact: High - Jobs marked as failed despite successful primary operations
• User Experience: Negative - Appears as complete failure when it's actually partial success

Configuration Issues (Cluster 2)

Primary Issue: Invalid secret configuration

The GitHub App private key is a critical security credential that must be:

• PEM-encoded RSA private key
• Include BEGIN/END PRIVATE KEY headers
• Properly formatted without extra whitespace or line breaks
• Kept secure and not exposed in logs

Impact Assessment:

• Functional Impact: Critical - Completely blocks push_to_pull_request_branch functionality
• Operational Impact: High - Workflows using this job fail immediately
• Security Impact: Medium - Invalid key prevents authentication (no security breach, but functionality broken)

Recommendations

Critical Priority

1. Fix GitHub App Private Key Configuration

Issue: Invalid or malformed private key prevents GitHub App authentication

Action:

# Verify the GH_APP_PRIVATE_KEY secret contains valid PEM format:
# -----BEGIN RSA PRIVATE KEY-----
# [base64 encoded key data]
# -----END RSA PRIVATE KEY-----

# To regenerate:
# 1. Go to GitHub App settings
# 2. Generate new private key
# 3. Update GH_APP_PRIVATE_KEY secret with complete key (including headers)
# 4. Test with a manual workflow run

Expected Outcome: push_to_pull_request_branch jobs will successfully authenticate

High Priority

2. Grant Token Permission for Issue Assignment

Issue: Cannot assign issues using GraphQL replaceActorsForAssignable mutation

Options:

Option A: Use GitHub App token instead of GITHUB_TOKEN

permissions:
  issues: write
  # Add GitHub App authentication
steps:
  - uses: actions/create-github-app-token@v1
    with:
      app-id: ${{ secrets.GH_APP_ID }}
      private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}

Option B: Make assignment optional (graceful degradation)

try {
  // Attempt to assign
  await assignIssue(issueNumber, assignee);
  core.info(`✓ Assigned issue #${issueNumber} to ${assignee}`);
} catch (error) {
  core.warning(`Could not assign issue: ${error.message}`);
  // Don't fail the job - primary operation succeeded
}

Recommended: Option B for immediate fix, Option A for long-term solution

3. Grant Token Permission for PR Reviewer Requests

Issue: Cannot request PR reviewers via REST API

Options:

Option A: Use GitHub App token (same as above)

Option B: Make reviewer request optional (graceful degradation)

try {
  // Attempt to request reviewers
  await requestReviewers(prNumber, reviewers);
  core.info(`✓ Requested reviewers for PR #${prNumber}`);
} catch (error) {
  core.warning(`Could not request reviewers: ${error.message}`);
  // Don't fail the job - PR was created successfully
}

Recommended: Option B for immediate fix, Option A for long-term solution

Medium Priority

4. Implement Graceful Degradation Pattern

Issue: Jobs fail completely when secondary operations fail, masking successful primary operations

Solution: Refactor safe output jobs to separate primary and secondary operations

// Primary operation - MUST succeed
const result = await createIssue(data);
core.setOutput('issue_number', result.number);
core.setOutput('issue_url', result.url);

// Secondary operations - OPTIONAL (log warnings on failure)
await attemptSecondaryOperations(result.number, {
  assign: data.assignee,
  addLabels: data.additionalLabels,
  linkToParent: data.parent
});

// Job succeeds if primary operation succeeds

Benefits:

• Clear distinction between required and optional operations
• Better visibility into what actually failed
• Improved success metrics
• Better user experience

5. Add Permission Validation Step

Issue: Jobs fail at runtime due to missing permissions

Solution: Add pre-flight permission check

async function validatePermissions() {
  const required = ['issues:write', 'pulls:write'];
  const optional = ['assignees:write', 'reviewers:write'];
  
  for (const perm of required) {
    if (!hasPermission(perm)) {
      core.setFailed(`Missing required permission: ${perm}`);
      return false;
    }
  }
  
  for (const perm of optional) {
    if (!hasPermission(perm)) {
      core.warning(`Missing optional permission: ${perm} - some features may not work`);
    }
  }
  
  return true;
}

Historical Context

Comparing with previous audit (2025-11-22):

Metric	2025-11-22	2025-11-23	Change
Total Failures	12	5	⬇️ -58%
Success Rate	88.9%	90.6%	⬆️ +1.7pp
Permission Errors	6	4	⬇️ -33%
Config Errors	1	1	➡️ Same

Trends

• Positive: Overall failure rate is decreasing
• Positive: Permission errors are reducing (likely due to fewer workflows running)
• Concern: Same configuration error persists (GitHub App key issue)
• Concern: Issue assignment failures remain unresolved

Most Common Recurring Issue

GitHub API Permission Errors continue to be the most common failure pattern, affecting both create_issue and create_pull_request jobs. This has been a recurring issue since at least 2025-11-20.

Work Item Plans

Work Item 1: Graceful Degradation for Secondary Operations

Type: Enhancement
Priority: High
Effort: Medium

Description: Refactor safe output jobs to treat secondary operations (assign, request reviewers) as optional, allowing jobs to succeed when primary operations complete successfully.

Acceptance Criteria:

• Jobs succeed when primary operation (create issue/PR) succeeds
• Secondary operation failures logged as warnings, not errors
• Job outputs reflect which operations succeeded/failed
• Step summary clearly shows primary vs secondary operation status

Technical Approach:

1. Wrap secondary operations in try-catch blocks
2. Log warnings instead of throwing errors
3. Update job conclusion logic to only fail on primary operation failures
4. Enhance step summary to show operation-by-operation status

Files to Modify:

• .github/workflows/safe-outputs/create-issue.yml
• .github/workflows/safe-outputs/create-pull-request.yml

Work Item 2: Fix GitHub App Private Key

Type: Bug Fix
Priority: Critical
Effort: Small

Description: Regenerate and update the GitHub App private key secret to restore push_to_pull_request_branch functionality.

Acceptance Criteria:

• New private key generated from GitHub App settings
• Secret updated in repository settings
• Test workflow run succeeds with new key
• No authentication errors in logs

Technical Approach:

1. Access GitHub App settings page
2. Generate new private key
3. Copy complete key including BEGIN/END headers
4. Update GH_APP_PRIVATE_KEY repository secret
5. Trigger test workflow run
6. Verify authentication succeeds

Dependencies: Access to GitHub App settings and repository secrets

Work Item 3: Add Permission Documentation

Type: Documentation
Priority: Medium
Effort: Small

Description: Document required and optional permissions for safe output jobs to help users troubleshoot permission errors.

Acceptance Criteria:

• Document lists all permissions used by each safe output job
• Document explains difference between required and optional permissions
• Document includes troubleshooting guide for common permission errors
• Document includes examples of proper token configuration

Technical Approach:

1. Audit each safe output job's API calls
2. Map API calls to required permissions
3. Create permission matrix table
4. Add troubleshooting section with common errors
5. Include example configurations

Location: docs/reference/safe-outputs.md

Next Steps

Immediate Actions (This Week)

1. ✅ Generate and share this health report
2. ⚠️ Regenerate GitHub App private key - Critical blocker for push_to_pull_request_branch
3. ⚠️ Implement graceful degradation - High impact on success metrics

Short Term (Next 2 Weeks)

1. 📋 Document permission requirements - Help users avoid permission errors
2. 🔍 Add permission validation - Catch issues early with clear messages
3. 📊 Track trends over 14 days - Confirm improvements are sustained

Long Term (Next Month)

1. 🔄 Migrate to GitHub App tokens - Comprehensive solution for permission issues
2. 📈 Automate trend analysis - Add charts and visualizations to reports
3. 🧪 Add integration tests - Test safe output jobs in isolation

Metrics and KPIs

• Overall Safe Output Success Rate: 90.6% (target: >95%)
• Most Reliable Job Type: create_discussion, add_comment, missing_tool (100% success)
• Most Problematic Job Type: create_issue (57% success, 3 failures)
• Failure Rate Trend: Improving (11.1% → 9.4%, -1.7pp)
• Time to Failure: N/A (jobs fail immediately at secondary operation)
• Critical Issues: 1 (GitHub App key)
• High Issues: 2 (permission errors)
• Medium Issues: 1 (graceful degradation)

---

References:

• §19516274301
• §19551191192
• §19590718399

AI generated by Safe Output Health Monitor

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.