[prompt-analysis] Copilot PR Prompt Analysis - 2025-12-08

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (ending 2025-12-08)
Dataset: 1000 Copilot-generated PRs
Overall Success Rate: 78.9%

Status	Count	Percentage
Merged	789	78.9%
Closed (not merged)	208	20.8%
Open	3	0.3%

Full Analysis Report

Prompt Categories and Success Rates

The analysis categorized prompts based on keywords in titles and descriptions. Categories are not mutually exclusive—a single PR can belong to multiple categories.

Category	Total PRs	Merged	Closed	Success Rate
Update	742	603	139	81.3%
Testing	750	608	142	81.1%
Refactor	387	311	76	80.4%
Feature	961	766	195	79.7%
Bug Fix	983	782	201	79.6%
Documentation	821	644	177	78.4%

Key Category Insights

• Update PRs have the highest success rate at 81.3%, suggesting dependency updates and version bumps are well-received
• Testing PRs follow closely at 81.1%, indicating test additions are valued
• Documentation has the lowest rate at 78.4%, though still respectable
• All categories show success rates above 78%, indicating generally high quality across the board

Prompt Length Analysis

Merged PRs: Average 471 words
Closed PRs: Average 424 words

The data shows merged PRs have slightly longer prompts on average, suggesting more detailed explanations correlate with success. However, the difference is modest (about 48 words), indicating quality matters more than quantity.

Keyword Analysis

Most Common Keywords Overall

In Merged PRs:

• copilot: 5959 occurrences (7.55 per PR)
• agent: 4548 occurrences (5.76 per PR)
• github: 3858 occurrences (4.89 per PR)
• workflow: 3740 occurrences (4.74 per PR)
• coding: 3611 occurrences (4.58 per PR)
• details: 2649 occurrences (3.36 per PR)
• summary: 2411 occurrences (3.06 per PR)
• https: 2298 occurrences (2.91 per PR)
• githubnext: 2014 occurrences (2.55 per PR)
• pkg: 1963 occurrences (2.49 per PR)

In Closed PRs:

• copilot: 1572 occurrences (7.56 per PR)
• agent: 1159 occurrences (5.57 per PR)
• coding: 938 occurrences (4.51 per PR)
• github: 828 occurrences (3.98 per PR)
• workflow: 789 occurrences (3.79 per PR)
• details: 654 occurrences (3.14 per PR)
• summary: 604 occurrences (2.90 per PR)
• https: 552 occurrences (2.65 per PR)
• mcp: 543 occurrences (2.61 per PR)
• add: 477 occurrences (2.29 per PR)

Distinctive Patterns

Keywords that appear significantly more often in one outcome vs another can reveal success patterns.

Keywords MORE common in MERGED PRs:

• cli: 1.6x more frequent in merged PRs

No distinctive keywords found that strongly correlate with closed PRs.

Example Prompts

✅ Successful (Merged) Prompts

1. PR #5820: Add early compilation explanation to Quick Start guide

User testing revealed confusion about compilation—users wondered why it's needed when regular GitHub Actions don't require compilation, what gets compiled, and whether they need to install a compiler....

Prompt length: 729 words

2. PR #5818: Add minItems constraint to schedule array in workflow schema

Empty schedule arrays (schedule: []) pass validation but create workflows that never run. This adds a minItems: 1 constraint to reject them at compile time. ## Changes - Schema: Added `"minIte...

Prompt length: 694 words

3. PR #5819: Document branches/paths mutual exclusivity in workflow schema

Adds $comment fields to main_workflow_schema.json documenting GitHub Actions' mutual exclusivity constraints: branches/branches-ignore and paths/paths-ignore cannot be used together. ## Ch...

Prompt length: 902 words

❌ Unsuccessful (Closed) Prompts

1. PR #5825: Verify strict field already exists in schema

Issue #5804 requested adding the strict field definition to the workflow schema to enable IDE autocomplete and validation. Investigation revealed the field was already added in commit 2e19f09. ## Cu...

Prompt length: 846 words

2. PR #5810: Hardcode safeinputs MCP server to port 52000 and remove API key authentication

Fix safeinputs MCP server configuration for Copilot engine Apply the same strategy used for GitHub "remote" MCP configuration to safeinputs: ### Changes Implemented:

Prompt length: 698 words

3. PR #5801: [WIP] Fix environment variable expansion for safeinputs MCP server

Plan: Fix Environment Variable Expansion in Safe-Inputs MCP Server Configuration The generated MCP configuration for the safeinputs server using Copilot contains unexpanded environment variables (`...

Prompt length: 294 words

Key Insights and Patterns

What Makes Prompts Successful?

1. Consistent Quality: With 78-81% success rates across all categories, most Copilot PRs are well-formed
2. Detail Matters: Merged PRs average ~20% more words than closed ones, suggesting thorough explanations help
3. Update PRs Excel: Dependency updates and version bumps have the highest success rate (81.3%)
4. Testing is Valued: PRs that add tests have 81.1% success rate, showing the project values test coverage

Common Characteristics of Closed PRs

Based on the examples analyzed:

• No-op changes: PRs that discover the work is already done (e.g., PR Verify strict field already exists in schema #5825, No changes needed: "date" already in DefaultBashTools since September 2025 #5800)
• Work-in-progress: PRs marked as [WIP] or exploratory (e.g., PR [WIP] Fix environment variable expansion for safeinputs MCP server #5801)
• Reverted changes: PRs that undo previous work after feedback (e.g., PR Revert Docker container implementation for safe-inputs MCP server #5784)
• Configuration issues: PRs attempting complex infrastructure changes that need iteration

These are often closed not due to poor prompts, but because they're superseded, obsolete, or part of an iterative process.

Recommendations

✅ DO: Best Practices for Copilot Prompts

1. Be specific about the change: Reference specific files, functions, or issues
2. Provide context: Explain why the change is needed, not just what to change
3. Include testing considerations: Mention how the change should be tested
4. Aim for 400-500 words: Detailed enough to be clear, concise enough to stay focused
5. Reference issues/PRs: Link to related issues or previous discussions

⚠️ CONSIDER: Factors That May Lead to Closure

1. Exploratory work: Mark PRs as [WIP] if you're not sure the approach will work
2. Complex infrastructure: Changes to build systems or CI may need multiple iterations
3. Check existing code first: Verify the feature/fix isn't already implemented
4. Configuration changes: MCP servers, Docker setups, etc. often need refinement

💡 INSIGHT: High Success Rate Indicates Mature Workflow

The 78.9% overall merge rate suggests:

• The Copilot agent is well-configured for this repository
• Most generated PRs meet quality standards
• The team has effective review processes
• Closed PRs are often due to external factors (obsolescence, iteration) rather than poor quality

Historical Trends

Date	Total PRs	Merged	Success Rate
2025-12-08	1000	789	78.9%
2025-12-06	1000	796	79.6%
2025-12-04	1000	786	78.6%
2025-12-03	1000	796	79.6%
2025-12-07	1000	787	78.7%

Trend: 📈 Success rate is improving (78.9% today vs 78.7% seven days ago)

---

References:

• §5820
• §5818
• §5825

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot PR Prompt Pattern Analysis.
🗺️ A newer treasure map awaits ye at Discussion #5933.
Fair winds, matey! 🏴‍☠️