[prompt-analysis] Copilot PR Prompt Analysis - November 19, 2025

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - November 19, 2025

This analysis examines 1,000 Copilot-generated PRs from the last 30 days in githubnext/gh-aw, identifying patterns that correlate with successful merges versus closed PRs.

Key Findings

Success Rate: 76.9% - Copilot PRs show a strong merge rate, with 684 of 890 completed PRs merged (206 closed without merge, 2 still open).

The analysis reveals that prompt clarity and action verb choice are strong indicators of success. PRs starting with "update", "fix", and "remove" show higher merge rates, while overly short prompts (< 5 words) or vague directives correlate with closures.

Detailed Analysis Report

Summary Statistics

Analysis Period: Last 30 days
Total PRs Analyzed: 892 PRs with extractable prompts
Completed PRs: 890 | Merged: 684 (76.9%) | Closed: 206 (23.1%)
Still Open: 2

Prompt Categories and Success Rates

Analysis of prompt types shows refactoring and documentation changes have the highest success rates:

Category	Total PRs	Merged	Success Rate
Refactoring	10	9	90.0%
Documentation	40	34	85.0%
Removal	33	28	84.8%
Updates/Modifications	167	135	80.8%
Bug Fixes	83	66	79.5%
Other/Uncategorized	480	366	76.2%
Feature Additions	180	130	72.2%
Testing	7	5	71.4%

Key Insight: Refactoring prompts show a 90% success rate, while feature additions and tests have lower rates (~71-72%), suggesting that well-scoped maintenance tasks are more likely to succeed than new functionality additions.

Prompt Characteristics Analysis

Length Analysis

Merged PRs: Average 10.2 words (median: 7 words)
Closed PRs: Average 10.7 words (median: 9 words)

Insight: Both successful and unsuccessful prompts average around 10 words. Length alone is not a strong predictor, but prompts under 5 words show higher failure rates.

Action Verb Analysis

Top verbs in MERGED PRs:

1. update (104 PRs)
2. add (71 PRs)
3. fix (38 PRs)
4. create (25 PRs)
5. remove (19 PRs)

Top verbs in CLOSED PRs:

1. update (28 PRs)
2. add (25 PRs)
3. create (17 PRs)
4. fix (7 PRs)
5. make (4 PRs)

Insight: "update" is the most common verb in both categories, but merged PRs show more diverse action verbs. The ratio of "fix" prompts is higher in merged PRs (38 vs 7), suggesting bug fixes are well-suited for Copilot.

Specificity Indicators

File references (mentions of .go, .js, file paths):

• Merged PRs: 13.5%
• Closed PRs: 10.7%

Code references (backticks, function names, code patterns):

• Merged PRs: 7.2%
• Closed PRs: 4.9%

Insight: PRs with specific file or code references show slightly higher merge rates (+2-3%), suggesting that specificity helps but is not required for success.

Example Prompts

✅ Successful Prompts (Merged)

1. Short and clear (PR #4311, 10 words)

"Make sure the bundler is applied to the no_op JavaScript"

• Result: Merged
• Why it worked: Clear action (apply bundler), specific component (no_op JavaScript)

2. Medium with file reference (PR #4297, 23 words)

"Update the custom agent that helps to create custom safe output type to specify to update the tools.json with the new tool signature."

• Result: Merged
• Why it worked: Specific file (tools.json), clear purpose (update tool signature)

3. Update directive (PR #4304, 10 words)

"Rename "Status" documentation page and generator to "Labs""

• Result: Merged
• Why it worked: Simple rename operation, specific component names

❌ Unsuccessful Prompts (Closed)

1. Too vague (PR #4300, 2 words)

"Lint code"

• Result: Closed (marked as [WIP])
• Issue: No scope, no specific files, overly broad

2. Complex bug fix (PR #4237, 12 words)

"Fix error rendering such that nested errors are shown on new lines."

• Result: Closed
• Issue: Ambiguous implementation details (how should they be shown?)

Key Insights

Based on pattern analysis across 892 prompts:

1. Refactoring > Features: Refactoring and documentation updates (85-90% success) outperform feature additions (72% success), likely because they have clearer success criteria.

2. Action verb matters: "update", "fix", and "remove" appear frequently in merged PRs, while closed PRs show more "create" and "add" (which may be more complex).

3. Specificity helps but isn't required: Only 13.5% of merged PRs explicitly mention files, suggesting Copilot can work effectively with general descriptions when the action is clear.

4. Sweet spot length: 7-10 words appears optimal - enough to be clear, concise enough to avoid ambiguity.

5. Bug fixes succeed: 79.5% success rate for bug fixes suggests Copilot excels at corrective tasks with clear problems to solve.

Recommendations

✅ DO:

1. Use clear action verbs: Start with "update", "fix", "remove", "rename" - these correlate with higher success
2. Be concise but complete: Aim for 7-15 words that clearly state the action and target
3. Choose maintenance over new features: Refactoring, fixes, and updates have higher success rates than new features
4. State the "what" clearly: Even without file names, clear descriptions of what to change work well

❌ AVOID:

1. Ultra-short prompts: Avoid 1-3 word prompts like "Lint code" - too vague
2. Overly complex features: Large feature additions have lower success rates
3. Ambiguous outcomes: "Improve" or "enhance" without specifics lead to uncertainty
4. Missing context: If the change requires understanding complex business logic, provide key details

💡 BEST PRACTICES:

Template for high-success prompts:

[Action verb] [component/file] to [outcome/purpose]

Examples:

• ✅ "Update error handling in parser to show line numbers"
• ✅ "Fix race condition in cache invalidation logic"
• ✅ "Remove deprecated webhook validation function"
• ❌ "Improve performance"
• ❌ "Add features"

Historical Trends

Date	PRs	Success Rate	Top Category
2025-11-18	1000	77.4%	bug_fix
2025-11-18	1000	77.0%	update
2025-11-19	892	76.9%	other

Trend: Success rate ↓ 0.1 percentage points from previous analysis

Summary

Copilot PRs show a strong 76.9% merge rate when prompts are clear and action-oriented. The data suggests that:

• Refactoring and maintenance tasks are ideal candidates for Copilot automation
• Clear, concise prompts (7-15 words) with specific action verbs work best
• Bug fixes and updates significantly outperform new feature creation
• Specificity helps but general descriptions work when the action is unambiguous

The patterns suggest developers should favor focused, single-purpose prompts over broad directives, and choose corrective/maintenance tasks over complex feature additions for highest success rates.

---

Generated by Copilot PR Prompt Analysis Agent

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.