[prompt-analysis] Copilot PR Prompt Analysis - 2025-12-09

[github-actions[bot]]

Analysis of 1,000 Copilot-generated PRs from the last 30 days reveals that successful prompts focus on workflow-level features and GitHub integration (78.8% merge rate), while unsuccessful prompts tend to deal with internal implementation details. The type of task matters less than how the prompt is written - all categories show 72-83% success rates.

Key finding: Bug fix prompts have the highest success rate (82.5%) when they include clear problem statements, while documentation-only and vague "suggestions" prompts correlate with closed PRs.

Full Analysis Report

Analysis Period

Last 30 days (Data as of 2025-12-09)

Summary Statistics

Metric	Count	Percentage
Total PRs	1,000	100%
Merged	788	78.8%
Closed (not merged)	209	20.9%
Open	3	0.3%

Completed PRs: 997 (merged + closed)

Prompt Categories and Success Rates

Analysis based on 580 PRs with extractable prompts (460 merged, 120 closed).

Category	Total	Merged	Closed	Success Rate
Bug Fix	126	104	22	82.5% ⭐
Update	174	141	33	81.0%
Documentation	78	63	15	80.8%
Refactoring	28	22	6	78.6%
Testing	31	24	7	77.4%
Feature	232	177	55	76.3%
Uncategorized	140	101	39	72.1%

Key Insight: All categories show similar success rates (72-83%), suggesting task type matters less than prompt quality.

Prompt Characteristics Analysis

Length Analysis

Metric	Merged PRs	Closed PRs
Avg Word Count	41.5 words	41.7 words
File References	54.6%	52.5%
Code Snippets	7.6%	9.2%

Finding: Prompt length and file references don't strongly correlate with success.

Action Verbs Distribution

Merged PRs (Top 5):

1. update - 134 occurrences (29.1%)
2. add - 129 occurrences (28.0%)
3. fix - 55 occurrences (12.0%)
4. create - 41 occurrences (8.9%)
5. remove - 28 occurrences (6.1%)

Closed PRs (Top 5):

1. add - 46 occurrences (38.3%)
2. update - 27 occurrences (22.5%)
3. create - 15 occurrences (12.5%)
4. remove - 10 occurrences (8.3%)
5. fix - 9 occurrences (7.5%)

Keyword Pattern Analysis

Keywords More Common in MERGED PRs

Keyword	Merged Count	Closed Count	Multiplier
logs	75	7	×2.8
run	122	13	×2.4
workflows	89	12	×1.9
issues	80	11	×1.9
safe	145	21	×1.8
agentic	152	24	×1.7
runs	99	16	×1.6
github	306	52	×1.5

Pattern: Successful prompts focus on workflow-level features and GitHub integration points.

Keywords More Common in CLOSED PRs

Keyword	Merged Count	Closed Count	Multiplier
suggestions	1	26	×99.7
pattern	9	23	×9.8
engine	18	25	×5.3
string	20	24	×4.6
documentation	20	22	×4.2
error	37	31	×3.2

Pattern: Unsuccessful prompts deal with internal implementation details and exploratory requests.

Example Prompts

✅ Successful (Merged)

1. PR #5918 - Add compile-time validation for script runtime

Add checks at compile time in the script registry: - GitHub-script should not use "execSync", exec instead - node scripts should not use github-script globals like core.* or exec.* or github.*

2. PR #5912 - Add labels field for workflow categorization

Add support for a top level "labels" field (takes an array of string) that allows to label an agentic workflow. The labels are not used currently, but they are reported in the "status/list" command.

3. PR #5911 - Add minimize-comment safe output type

Create "minimize-comment" safe output type that allows the agent to minimize a comment. - implemented with graphql

4. PR #5908 - Implement dependency graph for compile --watch

The "compile --watch" should build the graph of workflows and imports such that it always recompile all the dominator top agentic workflows.

5. PR #5907 - Implement AI moderator workflow

Create an agentic workflow that implements the same features as https://github.com/github/ai-moderator

❌ Unsuccessful (Closed)

1. PR #5913 - Add Absurd durable execution spec

Create a spec markdown file about absurd (specs/absurd.md) https://github.com/earendil-works/absurd Keep it short, prefer links.

2. PR #5903 - Add api.github.com network domain

Update the gh.md agentic workflow to allow network domain "api.gitbub.com"

3. PR #5888 - Add --deny-tool option

Update copilot agentic engine to deny tool "fetch_copilot_cli_documentation" by default

4. PR #5810 - Hardcode safeinputs MCP server port

Fix environment variable expansion/configuration of the safeinputs MCP server for the copilot engine - pre-assign port 3002 and remove dynamic port allocation

5. PR #5801 - Fix environment variable expansion

The generated configuration for the safeinputs mcp server using copilot is incorrect. The environment variables are not expanded (GH_AW_SAFE_INPUTS_PORT)

Key Insights

1. 🎯 Task Type Matters Less Than Prompt Quality

All categories show 72-83% success rates - the consistency suggests that how you write the prompt is more important than what type of task you're requesting.

2. 📏 Length ≠ Success

Both merged and closed PRs average ~41-42 words. Conciseness and verbosity don't correlate with outcomes.

3. 🔗 Workflow-Level Focus = Higher Success

Prompts mentioning workflows, runs, logs, issues, github, and agentic are 1.5-2.8× more common in merged PRs. These focus on user-facing workflow features rather than internal implementation.

4. 🐛 Bug Fixes Win with Clear Problem Statements

Bug fix prompts have the highest success rate (82.5%), suggesting that clear problem statements lead to better outcomes than exploratory or vague requests.

5. ⚙️ Internal Implementation = Lower Success

Keywords like engine, pattern, error, pkg, and documentation are 3-100× more common in closed PRs, indicating these prompts may be too implementation-focused or exploratory.

6. 📝 Documentation-Only Prompts Struggle

The documentation keyword is 4.2× more common in closed PRs, suggesting doc-only changes may not align with Copilot's code-generation strengths.

Recommendations

✅ DO

1. Focus on workflow-level features and GitHub integration

• Good: "Add workflow run analysis feature"
• Good: "Implement GitHub issue label filtering"
• Avoid: "Add pattern matching to internal engine"

2. Write clear problem statements for bug fixes

• Bug fixes have 82.5% success rate vs 76.3% for features
• Include error messages or reproduction steps
• Example: "Fix SC2086 shellcheck warnings in super-linter workflow"

3. Reference GitHub integration points

• Prompts with github, workflow, issues, runs more successful
• Example: "Update GitHub issue handling in workflow execution"

4. Be specific about user-facing impact

• Good: "Add labels field for workflow categorization and filtering"
• Avoid: "Update internal string pattern validation"

❌ AVOID

1. Prompts about internal engine/configuration details

• Keywords engine, error, pkg correlate with closed PRs
• These may be too implementation-focused for effective automation

2. Documentation-only requests

• documentation keyword 4.2× more common in closed PRs
• Consider combining docs with code changes

3. Vague suggestions or exploratory patterns

• suggestions keyword 100× more common in closed PRs
• Be specific about what to change, not just exploratory

4. Pure configuration fixes without clear user impact

• Example of closed PR: "Hardcode safeinputs MCP server to port 52000"
• These may be too narrow or implementation-specific

Historical Trends (Last 7 Days)

Date	PRs	Merged	Success Rate	Top Category
2025-12-09	1000	788	78.8%	feature
2025-12-08	1000	789	79.1%	feature
2025-12-06	1000	796	79.8%	feature
2025-12-04	1000	786	79.4%	feature
2025-12-03	1000	796	79.8%	feature
2025-12-07	1000	787	79.1%	feature

Trend: Success rate stable around 78-80% over the past week, with consistent patterns across days.

Prompt Writing Template

Based on this analysis, here's a template for high-success prompts:

[Action Verb] [Workflow-Level Feature] for [User-Facing Benefit]

Context:
- [Clear problem statement or user need]
- [Reference to GitHub integration point: issues, runs, workflows, etc.]
- [Specific files or components involved (optional)]

Expected Outcome:
- [Concrete deliverable]
- [How it improves the workflow experience]

Example:

Add workflow run cost analysis for budget monitoring

Context:
- Users need visibility into GitHub Actions minutes consumption
- Integrate with GitHub Actions API runs endpoint
- Display costs in gh aw audit command output

Expected Outcome:
- Add cost calculation to audit command
- Show per-workflow breakdown of minutes used

---

References:

• Analysis run: §20057957521
• Data: 1,000 Copilot PRs from last 30 days
• Cache: /tmp/gh-aw/cache-memory/prompt-analysis/history.json

AI generated by Copilot PR Prompt Pattern Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot PR Prompt Pattern Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6006.
Fair winds, matey! 🏴‍☠️