[prompt-analysis] Copilot PR Prompt Analysis - 2025-12-10

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2025-12-10

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 782 (78.5%) | Closed: 214 (21.5%) | Open: 4

The overall merge rate of 78.5% indicates that most Copilot-generated PRs successfully land, with key patterns emerging around prompt structure and specificity.

Full Analysis Details

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Test	825	660	80.0%
Refactor	369	295	79.9%
Feature	964	761	78.9%
Bug Fix	984	775	78.8%
Documentation	895	701	78.3%

Key Finding: Test prompts have the highest success rate at 80.0%, while Documentation prompts show the lowest at 78.3%. However, differences are minimal (< 2%), suggesting prompt type matters less than prompt quality.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 465 words
• Most common keywords: copilot, agent, coding, details, summary, workflow, start, github, command, tests
• Structure: Merged PRs tend to include clear problem statements, specific file references, and detailed change descriptions

Example successful prompts:

1. PR #5996: When /plan is called from an issue comment, it now uses that issue as the parent instead of creating a new tracking issue. Discussion-triggered `/pl... → Merged
2. PR #5995: The dev-hawk.md workflow used the deprecated agentic-workflows: tool configuration. Updated to use the standardized shared/mcp/gh-aw.md import p... → Merged
3. PR #5985: Adds compiler infrastructure to emit different action references based on environment context. Development workflows use local action paths (`./action... → Merged

Success Indicators:

• Specific problem identification
• Clear scope and file references
• Detailed change descriptions with rationale
• Technical context (error messages, logs, reproduction steps)

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 418 words
• Most common keywords: copilot, agent, coding, details, summary, start, workflow, tests, add, mcp
• Structure: Closed PRs often have "[WIP]" tags, generic descriptions, or lack specific implementation details

Example unsuccessful prompts:

1. PR #5998: ## Plan: Refine Created Items Display in Conclusion Comment

• Understand current implementation in add_comment.cjs
• Change format from `[#... → Closed

2. PR #5994: ## Safe Output Failure Reporting

Implement functionality to track and report safe output job failures:

• Understand current safe output job stru... → Closed

3. PR #5983: Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

(!-- ST... → Closed

Failure Indicators:

• "[WIP]" tags indicating incomplete work
• Generic task descriptions without specifics
• Missing technical context
• Vague scope ("fix issues", "update page")

Key Insights

Based on analysis of 1000 PRs over 30 days:

• Prompt Length: Merged PRs average 465 words vs 418 words for closed PRs. Longer, more detailed prompts correlate with success (+47 words).

• Specificity Matters: Merged PRs reference specific files, functions, and error messages. Closed PRs often use generic language ("fix issues", "update documentation").

• Technical Context: Successful prompts include error messages, reproduction steps, and code references. Failed prompts lack concrete details.

• WIP Anti-Pattern: PRs with "[WIP]" titles have significantly lower merge rates. These indicate incomplete planning or exploratory work.

Recommendations

Based on 30-day analysis of 1000 Copilot PRs:

✅ DO: Write Detailed, Specific Prompts

1. Include specific file paths and function names: "Fix error handling in pkg/cli/compile.go:CompileWorkflow()" vs "Fix compile errors"
2. Provide technical context: Include error messages, stack traces, or reproduction steps
3. Set clear scope: Define what should and shouldn't change
4. Add rationale: Explain why the change is needed and how it solves the problem

✅ DO: Structure Your Prompt

Problem: [Specific issue with error message/reproduction]
Files: [List specific files to modify]
Changes: [Clear description of what to change]
Testing: [How to verify the fix]

❌ AVOID: Vague or Incomplete Prompts

1. Don't use WIP approach: Plan fully before starting Copilot PR
2. Avoid generic descriptions: "Fix issues" → "Fix null pointer in workflow parser"
3. Don't omit context: Include relevant error messages and logs
4. Avoid scope creep: Focus on one clear objective per PR

Category Performance Summary

All prompt categories show similar success rates (78.3% - 80.0%), suggesting how you write the prompt matters more than what type of change you're making.

• Test prompts perform best (80.0%) - likely because test requirements are naturally specific
• Refactor prompts also succeed (79.9%) - clear scope helps
• Bug Fix and Feature prompts nearly identical (~79%) - prompt quality dominates

Historical Trends

First analysis - no historical data available yet.

Future reports will track:

• Week-over-week success rate changes
• Emerging prompt patterns
• Category performance trends

</details)

---

References:

• §20093218304

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot PR Prompt Pattern Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6115.
Fair winds, matey! 🏴‍☠️