[prompt-clustering] Copilot Agent Prompt Clustering Analysis - 2025-12-01

[github-actions[bot]]

🔬 Copilot Agent Prompt Clustering Analysis

Daily NLP-based clustering analysis of copilot agent task prompts

Analysis Date: 2025-12-01

Executive Summary

Analyzed 1,373 copilot agent tasks using NLP clustering techniques (TF-IDF + K-means).
The analysis identified 8 distinct task categories with an overall success rate of 75.3% (merged PRs).

Key Findings:

• Most Common Task Type: Documentation (408 tasks, 29.7%)
• Highest Success Rate: Bug Fix (84.1%)
• Average Code Changes: 16.5 files per task
• Success Distribution: 1034 merged, 289 closed, 50 open

Full Clustering Analysis Report

Overall Statistics

• Total Tasks Analyzed: 1,373
• Merged PRs: 1034 (75.3%)
• Closed PRs: 289 (21.0%)
• Open PRs: 50 (3.6%)
• Clusters Identified: 8
• Average Files Changed: 16.5

Methodology

NLP Pipeline:

1. Extracted task prompts from 1,406 copilot PR descriptions
2. Text preprocessing: removed code blocks, URLs, markdown formatting
3. TF-IDF vectorization: 150 features, 1-3 gram range
4. K-means clustering: 8 clusters (optimal via silhouette score)
5. Theme categorization: automated classification based on term frequency

Cluster Analysis

Cluster 1: Documentation

Size: 408 tasks (29.7% of total)

Success Metrics:

• Merged: 313 (76.7%)
• Closed: 79
• Open: 16

Code Change Profile:

• Avg Files Changed: 16.4
• Avg Additions: +735 lines
• Avg Deletions: -194 lines

Top Keywords: github, workflows, documentation, cli, command, mcp, copilot, code

Characteristics: This cluster represents documentation tasks.

Example PRs: #2097, #2099, #2100

---

Cluster 2: Bug Fix

Size: 233 tasks (17.0% of total)

Success Metrics:

• Merged: 196 (84.1%)
• Closed: 31
• Open: 6

Code Change Profile:

• Avg Files Changed: 10.9
• Avg Additions: +1267 lines
• Avg Deletions: -288 lines

Top Keywords: workflow, agentic, workflows, run, file, github, issue, agent

Characteristics: This cluster represents bug fix tasks.
High success rate (84.1%) indicates these tasks are well-suited for the agent.

Example PRs: #2103, #2104, #2107

---

Cluster 3: Update/Enhancement

Size: 202 tasks (14.7% of total)

Success Metrics:

• Merged: 145 (71.8%)
• Closed: 41
• Open: 16

Code Change Profile:

• Avg Files Changed: 32.7
• Avg Additions: +1945 lines
• Avg Deletions: -937 lines

Top Keywords: output, safe, test, add, tests, update, run, changes

Characteristics: This cluster represents update/enhancement tasks.

Example PRs: #2111, #2127, #2137

---

Cluster 4: Documentation

Size: 140 tasks (10.2% of total)

Success Metrics:

• Merged: 104 (74.3%)
• Closed: 34
• Open: 2

Code Change Profile:

• Avg Files Changed: 12.5
• Avg Additions: +597 lines
• Avg Deletions: -180 lines

Top Keywords: copilot, servers, docs, mcp servers, aw make copilot, aw make, gh aw make, model context protocol

Characteristics: This cluster represents documentation tasks.

Example PRs: #2209, #2283, #2284

---

Cluster 5: General Development

Size: 135 tasks (9.8% of total)

Success Metrics:

• Merged: 104 (77.0%)
• Closed: 30
• Open: 1

Code Change Profile:

• Avg Files Changed: 12.0
• Avg Additions: +528 lines
• Avg Deletions: -251 lines

Top Keywords: copilot, thoughts, survey, thoughts copilot coding, share thoughts copilot, thoughts copilot, input share, love

Characteristics: This cluster represents general development tasks.

Example PRs: #2128, #2347, #2382

---

Cluster 6: Performance

Size: 129 tasks (9.4% of total)

Success Metrics:

• Merged: 105 (81.4%)
• Closed: 20
• Open: 4

Code Change Profile:

• Avg Files Changed: 12.5
• Avg Additions: +283 lines
• Avg Deletions: -176 lines

Top Keywords: set, coding, coding agent, copilot, agent, works faster, let copilot coding, higher quality

Characteristics: This cluster represents performance tasks.
High success rate (81.4%) indicates these tasks are well-suited for the agent.

Example PRs: #2254, #2282, #2285

---

Cluster 7: Documentation

Size: 78 tasks (5.7% of total)

Success Metrics:

• Merged: 63 (80.8%)
• Closed: 12
• Open: 3

Code Change Profile:

• Avg Files Changed: 22.2
• Avg Additions: +1289 lines
• Avg Deletions: -1506 lines

Top Keywords: files, workflow, github, command, documentation, file, workflows, agentic

Characteristics: This cluster represents documentation tasks.
High success rate (80.8%) indicates these tasks are well-suited for the agent.

Example PRs: #2108, #2109, #2151

---

Cluster 8: Feature Addition

Size: 48 tasks (3.5% of total)

Success Metrics:

• Merged: 4 (8.3%)
• Closed: 42
• Open: 2

Code Change Profile:

• Avg Files Changed: 1.5
• Avg Additions: +115 lines
• Avg Deletions: -92 lines

Top Keywords: make, work, changes, add, code, works faster, work set, use

Characteristics: This cluster represents feature addition tasks.
Lower success rate (8.3%) suggests these tasks may be more challenging.

Example PRs: #2101, #2110, #2126

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Top Keywords
1	Bug Fix	233	84.1%	10.9	workflow, agentic, workflows
3	Performance	129	81.4%	12.5	set, coding, coding agent
7	Documentation	78	80.8%	22.2	files, workflow, github
4	General Development	135	77.0%	12.0	copilot, thoughts, survey
6	Documentation	408	76.7%	16.4	github, workflows, documentation
5	Documentation	140	74.3%	12.5	copilot, servers, docs
2	Update/Enhancement	202	71.8%	32.7	output, safe, test
8	Feature Addition	48	8.3%	1.5	make, work, changes

Sample Task Data

Representative sample of tasks from each cluster:

PR #	Cluster	Theme	Status	Keywords
#2097	6	Documentation	✅ Merged	github, workflows, documentation
#2099	6	Documentation	✅ Merged	github, workflows, documentation
#2100	6	Documentation	✅ Merged	github, workflows, documentation
#2102	6	Documentation	❌ Closed	github, workflows, documentation
#2112	6	Documentation	❌ Closed	github, workflows, documentation
#2103	1	Bug Fix	✅ Merged	workflow, agentic, workflows
#2104	1	Bug Fix	✅ Merged	workflow, agentic, workflows
#2107	1	Bug Fix	✅ Merged	workflow, agentic, workflows
#2118	1	Bug Fix	❌ Closed	workflow, agentic, workflows
#2120	1	Bug Fix	❌ Closed	workflow, agentic, workflows
#2111	2	Update/Enhancement	❌ Closed	output, safe, test
#2127	2	Update/Enhancement	✅ Merged	output, safe, test
#2137	2	Update/Enhancement	✅ Merged	output, safe, test
#2145	2	Update/Enhancement	❌ Closed	output, safe, test
#2156	2	Update/Enhancement	❌ Closed	output, safe, test
#2209	5	Documentation	✅ Merged	copilot, servers, docs
#2283	5	Documentation	✅ Merged	copilot, servers, docs
#2284	5	Documentation	✅ Merged	copilot, servers, docs
#2293	5	Documentation	❌ Closed	copilot, servers, docs
#2310	5	Documentation	❌ Closed	copilot, servers, docs
#2128	4	General Development	❌ Closed	copilot, thoughts, survey
#2347	4	General Development	✅ Merged	copilot, thoughts, survey
#2382	4	General Development	✅ Merged	copilot, thoughts, survey
#2384	4	General Development	❌ Closed	copilot, thoughts, survey
#2402	4	General Development	❌ Closed	copilot, thoughts, survey
#2254	3	Performance	❌ Closed	set, coding, coding agent
#2282	3	Performance	❌ Closed	set, coding, coding agent
#2285	3	Performance	✅ Merged	set, coding, coding agent
#2343	3	Performance	❌ Closed	set, coding, coding agent
#2367	3	Performance	❌ Closed	set, coding, coding agent
#2108	7	Documentation	✅ Merged	files, workflow, github
#2109	7	Documentation	✅ Merged	files, workflow, github
#2151	7	Documentation	✅ Merged	files, workflow, github
#2153	7	Documentation	❌ Closed	files, workflow, github
#2155	7	Documentation	❌ Closed	files, workflow, github
#2101	8	Feature Addition	❌ Closed	make, work, changes
#2110	8	Feature Addition	❌ Closed	make, work, changes
#2126	8	Feature Addition	✅ Merged	make, work, changes
#2187	8	Feature Addition	❌ Closed	make, work, changes
#2213	8	Feature Addition	❌ Closed	make, work, changes

Key Findings

1. Documentation Tasks Dominate

626 tasks (45.6%) across 3 clusters involve documentation work.
This includes updating documentation, adding guides, improving CLI command docs, and MCP server documentation.

2. Bug Fixes Have High Success Rates

Bug fix tasks achieve 84.1% success rate,
suggesting the agent is particularly effective at targeted fixes with clear objectives.

3. Complex Feature Additions Are Challenging

The 'Feature Addition' cluster has the lowest success rate at 8.3%.
Tasks in this cluster involve: make, work, changes, add, code.
This suggests that open-ended feature additions may need more specific requirements or guidance.

Recommendations

Based on the clustering analysis, here are actionable recommendations:

1. Optimize for High-Success Task Types

Focus on task types with proven success rates:

• Bug Fix (84.1% success): 233 successful examples
• Performance (81.4% success): 129 successful examples
• Documentation (80.8% success): 78 successful examples

2. Improve Prompts for Low-Success Clusters

Tasks in these categories need better-defined requirements:

• Feature Addition (8.3% success): Provide clearer acceptance criteria and examples
• Update/Enhancement (71.8% success): Provide clearer acceptance criteria and examples

3. Leverage Documentation Patterns

Since documentation tasks are most common and have solid success rates (76.7%),
create reusable prompt templates for:

• CLI command documentation updates
• README enhancements
• API documentation
• MCP server documentation

4. Break Down Complex Features

Tasks involving complex features show lower success rates. Consider:

• Breaking large features into smaller, focused tasks
• Providing more detailed technical specifications
• Including example code or reference implementations

---

Methodology: TF-IDF vectorization with K-means clustering (k=8)

Data Coverage: 1,373 copilot agent PRs from githubnext/gh-aw

---

References:

• §19834278045

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #5325.
Fair winds, matey! 🏴‍☠️