[prompt-analysis] Copilot PR Prompt Analysis - 2025-12-04

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2025-12-04

This analysis examined 1,000 Copilot-generated PRs from the last 30 days to identify patterns that correlate with successful merges versus closed PRs.

Executive Summary

Analysis Period: Last 30 days (Nov 4 - Dec 4, 2025)
Total PRs Analyzed: 1,000 | Merged: 786 (79.4%) | Closed: 204 (20.6%) | Open: 10

Key Finding: Successful prompts are 23% more detailed (average 476 words vs 387 words) and include more specific references to commands, tests, and GitHub actions.

Analysis Methodology

This analysis:

• Extracted prompt text from PR bodies
• Categorized PRs by type (bug fix, feature, documentation, etc.)
• Analyzed keyword frequency, action verbs, and prompt characteristics
• Compared patterns between merged and closed PRs
• Calculated success rates by category

Success Rates by Category

Category	Total	Merged	Closed	Success Rate
Update	747	605	142	81.0%
Refactor	428	345	83	80.6%
Test	809	649	160	80.2%
Feature	956	763	193	79.8%
Bug Fix	976	778	198	79.7%
Documentation	797	625	172	78.4%

Observations:

• Update prompts have the highest success rate (81.0%) - these are focused, specific changes
• Documentation prompts have the lowest success rate (78.4%) - but still strong at nearly 4 in 5 merged
• Success rates are remarkably consistent (78-81%), suggesting prompt quality matters more than category

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average length: 476 words (23% longer than closed PRs)
• File references: 86% mention specific files (vs 79% in closed PRs)
• Issue references: 98% link to issues/PRs (vs 90% in closed PRs)

Top keywords in merged PRs:

• copilot (888 occurrences)
• agent (649 occurrences)
• github (627 occurrences)
• coding (480 occurrences)
• https (432 occurrences)
• details (421 occurrences)
• workflow (368 occurrences)
• summary (355 occurrences)

Most common action verbs:

• Primary: fix (99), add (92), update (71)
• Secondary: test (80), document (50), resolve (49)

Example successful prompts:

Update - PR #5496

Update actions/setup-java@v4 SHA to match current tag

Prompt preview: The TestActionPinSHAsMatchVersionTags test was failing because the pinned SHA for actions/setup-java@v4 no longer matched the tag's current commit.

Changes

Updated SHA from c5195efe... to `...

Bug Fix - PR #5496

Update actions/setup-java@v4 SHA to match current tag

Prompt preview: The TestActionPinSHAsMatchVersionTags test was failing because the pinned SHA for actions/setup-java@v4 no longer matched the tag's current commit.

Changes

Updated SHA from c5195efe... to `...

Refactor - PR #5487

Move setup guide to install.md with init and workflow import options

Prompt preview: The setup guide has been moved from the setup-agentic-workflows agent to a standalone install.md file that users access via GitHub Copilot. The install script now provides instructions for completin...

❌ Patterns in Closed PRs

Common characteristics in closed PRs:

• Average length: 387 words (23% shorter than merged PRs)
• Less specific references: Fewer mentions of specific commands, tests, and actions
• Similar categories: Closed PRs come from same categories but with less detail

Keywords appearing MORE in closed PRs:

• No keywords significantly more common in closed PRs (merged PRs dominate across all keywords)

Note: The absence of keywords more common in closed PRs suggests that successful prompts are differentiated by what they include, not what they avoid.

🎯 Key Insights

1. Overall success rate: 79.4% of completed PRs are merged

2. Successful prompts are 89 words longer (22.9% more detailed)

3. 'update' prompts have highest success rate (81.0%)

4. 'documentation' prompts have lowest success rate (78.4%)

5. Specificity wins: Merged PRs are 23% more detailed and include:

   • Specific file names and paths
   • Command examples
   • Test references
   • Links to related issues/PRs

6. Context matters: Keywords like command (+111%), update (+83%), and test (+42%) appear significantly more in merged PRs, suggesting explicit technical context improves success

📋 Recommendations

Based on the analysis of 1,000 PRs:

DO ✅

1. Be specific and detailed: Aim for 400+ words in your prompt

   • Include specific file paths and names
   • Reference relevant commands or functions
   • Link to related issues or PRs

2. Include technical context:

   • Mention tests that need to pass
   • Reference specific error messages
   • Note which GitHub Actions or workflows are affected

3. Use clear action verbs: fix, add, update, test

   • These verbs correlate with higher success rates
   • Be explicit about what needs to change

4. Focus on updates and refactoring:

   • These categories have the highest success rates
   • Break large features into smaller update-focused tasks

AVOID ❌

1. Vague or general descriptions:

   • Closed PRs average 23% fewer words
   • Missing specific file or command references

2. Skipping issue/PR links:

   • 98% of merged PRs reference issues vs 90% of closed
   • Context helps Copilot make better decisions

3. Overly broad scope:

   • Multiple unrelated changes in one prompt
   • Complex feature additions without clear boundaries

📊 Historical Trends

Date	Total PRs	Merged	Success Rate	Top Category
2025-12-04	1000	786	79.4%	Update (81.0%)
2025-12-03	1000	796	79.6%	Test (80.9%)

Trend Observation: Success rate remains stable around 79-81%, indicating consistent prompt quality across time periods.

💡 Example Prompt Template

Based on successful patterns, here's a recommended template:

**Context**: [Brief description of the problem or feature]

**Related Issue**: #[issue_number]

**Changes Needed**:
1. [Specific file or component to modify]
2. [Another specific change]

**Files Involved**:
- `path/to/file1.go`
- `path/to/file2.js`

**Tests**: [Mention relevant tests that should pass]

**Expected Outcome**: [Clear success criteria]

This structure includes:

• Specific file references ✓
• Issue link ✓
• Clear action items ✓
• Test context ✓
• Sufficient detail (400+ words when expanded) ✓

---

Generated by Copilot PR Prompt Analysis
Workflow Run: §19923564544
Data Source: 1,000 Copilot PRs from the last 30 days

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot PR Prompt Pattern Analysis.
🗺️ A newer treasure map awaits ye at Discussion #5734.
Fair winds, matey! 🏴‍☠️