[prompt-analysis] Copilot PR Prompt Analysis - 2025-12-14

[github-actions[bot]]

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 776 (77.9%) | Closed: 220 (22.1%) | Open: 4

Executive Summary

Analysis of 1000 Copilot-generated pull requests over the past 30 days reveals that bug fixes dominate both volume and success rates. With a 77.9% overall merge rate, prompts that are specific, detailed, and error-focused significantly outperform vague or broad-scope requests.

Key Finding: Successful prompts average 449 words compared to 404 words for closed PRs—an 11% difference suggesting that detail and specificity matter.

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Bug Fix	983	769	214	78.2% ⭐
Documentation	3	3	0	100% 🎯
Feature Addition	9	4	5	44.4%
Testing	1	0	1	0%

Key Observations:

• Bug fixes are the workhorse: 98% of all PRs, with strong success rate
• Documentation changes have perfect success but low volume
• Feature requests are risky: less than 50% success rate
• Testing-only changes struggled in this dataset

Prompt Analysis Deep Dive

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 449.3 words
• Most common keywords: copilot (473), agent (247), github (163), workflow (143), coding (132)
• Action verbs: fix, resolve, update, implement
• Technical specificity: References to MCP, safe outputs, tests, workflows

Example successful prompts:

1. PR #6385 → ✅ Merged

   "## CI Optimization: Integration Test Rebalancing This PR optimizes the CI integration test matrix b..."

   • Length: 1,474 words
   • Category: Bug fix
   • Why it worked: Extremely detailed, specific technical scope, clear problem statement

2. PR #6382 → ✅ Merged

   "## Fix Network Firewall Codemod to Handle Nested Properties - [x] Fix YAML syntax errors in smoke-c..."

   • Length: 362 words
   • Category: Bug fix
   • Why it worked: Clear fix target, technical specificity, error references

3. PR #6378 → ✅ Merged

   "Agent conversation output in GitHub Actions step summaries was truncating at 50 lines, cutting off c..."

   • Length: 404 words
   • Category: Bug fix
   • Why it worked: Specific problem (truncation), clear context

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 404.2 words (11% shorter than successful)
• Most common keywords: copilot (158), agent (103), coding (68), github (38)
• Issues identified: Vague scope, automation-focused, broad changes

Example unsuccessful prompts:

1. PR #6386 → ❌ Closed

   "Automates security scanning (zizmor, poutine, actionlint) on PRs that modify workflow files to catch..."

   • Length: 464 words
   • Category: Feature addition
   • Why it failed: Broad automation request, multiple tools, unclear integration points

2. PR #6379 → ❌ Closed

   "Comprehensive validation confirms the repository is in a clean state with no outstanding issues. ##..."

   • Length: 290 words
   • Category: Bug fix
   • Why it failed: "Comprehensive" scope, validation-focused but unclear problem

3. PR #6376 → ❌ Closed

   "Poutine flagged 40+ occurrences of curl -fsSL (redacted) | sudo bash across 20+..."

   • Length: 484 words
   • Category: Bug fix
   • Why it failed: Large-scale change (40+ occurrences), security tool-driven

Key Insights and Patterns

📊 Statistical Findings

1. Detail Matters: Merged PRs average 45 words longer (11.1% more detail)
2. Category Success: Bug fixes (78.2%) >> Features (44.4%) >> Tests (0%)
3. Keyword Frequency: Successful PRs mention "workflow" 143x vs 22x in closed PRs
4. Volume Concentration: 98% of PRs are bug fixes, indicating repository maturity

🎯 What Makes Prompts Successful

Specificity wins:

• Reference specific files, functions, or error messages
• Include technical terms relevant to the codebase (MCP, workflow, safe outputs)
• Provide context about why the change is needed

Scoping matters:

• Single-issue fixes succeed more than multi-issue batches
• Documentation-only changes have perfect record
• Feature additions need very clear boundaries

Length correlation:

• Sweet spot appears to be 400-500 words
• Too short (< 300 words) = missing context
• Extremely long (1000+ words) can work if highly detailed

🚫 What Makes Prompts Fail

Vague scope:

• Terms like "comprehensive", "automate everything", "improve all"
• Multiple unrelated changes bundled together
• Unclear success criteria

Broad automation:

• "Automate security scanning across all workflows"
• "Add monitoring to everything"
• Tool-driven without clear integration plan

Insufficient detail:

• Short prompts without error messages or reproduction steps
• Missing technical context
• No reference to specific code locations

Recommendations for Writing Better Prompts

✅ DO: Best Practices

1. Focus on bug fixes (78.2% success rate)

   • Reference specific errors or issues
   • Include reproduction steps when relevant
   • Mention exact files or functions affected

2. Write detailed prompts (450+ words average for success)

   • Explain the problem clearly
   • Describe expected vs. actual behavior
   • Provide technical context

3. Include specific technical terms

   • Use repository-specific vocabulary (workflow, MCP, safe outputs)
   • Reference existing patterns in the codebase
   • Mention relevant tools and frameworks

4. Prioritize documentation changes (100% success rate)

   • Clear, focused documentation updates
   • Update docs alongside code changes
   • Fix documentation bugs/inconsistencies

5. Be specific about scope

   • Single issue per PR works best
   • Clear boundaries and acceptance criteria
   • Avoid bundling unrelated changes

❌ AVOID: Anti-Patterns

1. Very short prompts (< 300 words)

   • Insufficient context leads to misinterpretation
   • Missing technical details
   • Unclear success criteria

2. Feature requests without clear scope (44.4% success)

   • Define exact boundaries
   • Break large features into smaller PRs
   • Include acceptance criteria

3. Vague automation requests

   • "Comprehensive", "automate everything", "improve all"
   • Multiple tools without clear integration
   • Unclear how changes fit together

4. Test-only changes (low success in this dataset)

   • May need to pair with implementation
   • Consider bundling with related code changes

5. Broad sweeping changes

   • 40+ file changes often rejected
   • Large-scale refactors need more planning
   • Break into incremental steps

📝 Prompt Template for High Success

## [Clear, specific title describing the fix]

**Problem**: [Specific error, bug, or issue - include error messages]

**Context**: [Why this matters, what's broken, reproduction steps]

**Technical Details**:
- Files affected: [List specific files]
- Root cause: [What's causing the issue]
- Existing patterns: [How similar cases are handled]

**Proposed Solution**: [Clear description of the fix approach]

**Validation**: [How to verify the fix works]

Historical Trends

Trend Analysis

This is the first automated analysis run for this repository. Future reports will include week-over-week comparisons showing:

• Success rate trends over time
• Category distribution changes
• Prompt length evolution
• Keyword frequency shifts

Next run: Tomorrow at the same time

---

References:

• §20205684824

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot PR Prompt Pattern Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6496.
Fair winds, matey! 🏴‍☠️