[prompt-analysis] Copilot PR Prompt Analysis - 2026-03-23

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-02-22 → 2026-03-23)
Total PRs: 1,000 | Merged: 825 (82.5%) | Closed: 173 (17.3%) | Open: 2 (0.2%)

Overall success rate has been gradually improving over the past week (+2.3 pp), reaching 82.7% today.

---

Prompt Categories and Success Rates

Category	Total	Merged	Closed	Success Rate
Bug Fix	902	749	151	83.2%
Documentation	28	24	4	85.7%
Testing	54	40	14	74.1%
Feature Addition	13	9	4	69.2%
Refactoring	2	2	0	100% (small sample)

Note: Categories are assigned based on title/body keywords; bug_fix dominates because the codebase is in active development.

---

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average body length: ~911 words
• Dominant prefix: fix: (87.3% success rate) vs feat: (78.3%)
• Specific context: clear description of what broke and why
• Concise, targeted scope (single concern)
• Body length has minimal impact on success — all length buckets hover around 82–85%

Example successful prompts:

1. PR #22377: fix: use <strong> instead of ** bold in <summary> elements → Merged

   "Markdown **bold** syntax is not processed inside HTML elements, so the firewall warning summary rendered as literal **...**"

2. PR #22359: perf: eliminate hot-path regexp compilations and redundant YAML parses → Merged

   "BenchmarkCompileComplexWorkflow regressed +113.9% (6.5ms → 13.9ms) due to regexp.MustCompile calls inside hot-path..."

3. PR #22361: fix: remove required inputs from smoke-* workflows, always provide defaults → Merged

   "Removes required: true from all inputs in smoke-* workflows so they can be triggered manually..."

---

❌ Unsuccessful Prompt Patterns

The WIP signal is the strongest predictor of failure:

PR Type	Merged	Closed	Success Rate
WIP PRs	17	36	32.1%
Non-WIP PRs	808	137	85.5%

• 20.8% of all closed PRs are marked [WIP] vs only 2.1% of merged PRs
• PRs with checklists ([ ] items) have a slightly lower success rate (77.7% vs 84.3%), possibly because WIP PRs often include task checklists

Example unsuccessful prompts:

1. PR #22345: [WIP] Fix agent assignment comment behavior on failure → Closed

   "Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan..."

2. PR #22205: Fix CRITICAL toolName shell injection in mcp_setup_generator.go → Closed

   "Closed despite a detailed, urgent description — showing that urgency/priority labels don't guarantee merges

3. PR #22244: fix(compiler): use push-to-pull-request-branch target-repo for safe_outputs check → Closed

   Well-described PR with specific root cause; closure may reflect technical incompatibility or superseding fix

---

Key Insights

• Pattern 1 — WIP is a merge killer: PRs marked [WIP] have a 32.1% success rate — less than half the 85.5% baseline for non-WIP PRs. Avoid using WIP status unless genuinely exploratory; prefer feature branches for work-in-progress.

• Pattern 2 — Bug fixes dominate and succeed: 90.2% of all PRs are bug-fix oriented (reflecting the project's iterative development phase), and they have a strong 83.2% success rate. Concrete, scoped bug reports with reproduction context perform best.

• Pattern 3 — Feature and test PRs face more friction: Feature addition PRs (69.2%) and testing PRs (74.1%) have meaningfully lower success rates, suggesting these often require more iteration or discussion before merging.

---

Recommendations

1. DO: Start prompts with fix: and include a clear root cause — this prefix correlates with the highest success rate (87.3%)
2. DO: Keep prompts focused on a single concern; body length doesn't matter, but specificity does
3. DO: For documentation PRs, state exactly which files change and why — 85.7% success rate makes this the safest category
4. AVOID: Using [WIP] unless intentionally exploratory — it signals incomplete work and is strongly associated with closures (3× lower success)
5. AVOID: Vague urgency markers (e.g., "CRITICAL") without accompanying implementation clarity — they don't improve merge rates

---

Historical Trends (Last 7 Days)

Date	Total PRs (rolling)	Success Rate	Delta
2026-03-17	1,000	81.4%	—
2026-03-18	1,000	82.0%	+0.6 pp
2026-03-19	1,000	81.9%	-0.1 pp
2026-03-21	1,000	82.3%	+0.4 pp
2026-03-22	1,000	82.6%	+0.3 pp
2026-03-23	1,000	82.7%	+0.1 pp

Trend: Steady improvement over the past week (+1.3 pp). The gradual increase suggests improved prompt quality or faster triage of non-viable PRs.

Data Notes & Methodology

• Dataset: 1,000 most recent PRs from copilot/* branches (last 30 days)
• Categories assigned via keyword matching on PR title + body (bug_fix takes priority over testing/feature)
• "WIP" detection: title starts with [WIP] or WIP
• Success rate = merged / (merged + closed), excluding open PRs
• Historical trend uses rolling 30-day window snapshots stored in cache-memory

References:

• Workflow Run §23422430166

AI generated by Copilot PR Prompt Pattern Analysis · history

• expires on Mar 24, 2026, 5:16 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #22599.