[prompt-clustering] Copilot Agent Prompt Clustering Analysis - December 2025

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns, trends, and opportunities for optimization.

Summary

Analyzed 1,892 copilot agent tasks from the last 30 days using advanced NLP clustering techniques (TF-IDF + K-means). The analysis identified 6 distinct clusters representing different types of tasks with varying success rates and complexity characteristics.

Overall Performance:

• Success Rate: 74.5% (1,409 tasks merged)
• Closed (Not Merged): 401 tasks (21.2%)
• Still Open: 82 tasks (4.3%)
• Average Complexity: 18.4 files changed, +1,289/-535 lines

Key Findings

🏆 Most Common Task Type: New Features (868 tasks, 45.9%)
⭐ Highest Success Rate: New Features - Agentic Workflows (79.8%)
⚠️ Lowest Success Rate: CI/CD & Workflows (65.8%)
📊 Average Code Changes: +1,289/-535 lines per task

Full Cluster Analysis

Cluster Analysis

Cluster 1: New Features - General

Size: 868 tasks (45.9% of total)

Performance:

• Success Rate: 72.7% (631 merged)
• Closed (not merged): 189
• Still open: 48

Characteristics:

• Avg Files Changed: 21.9
• Avg Additions: 1,002 lines
• Avg Comments: 2.1
• Avg Reviews: 1.3

Top Keywords: add, update, agent, javascript, run, github, step

Example Tasks:

• #2097: Add minimal path format syntax reference to imports documentation

  Update the frontmatter "imports" documentation under /docs with all the supported URL and path forma...

• #2099: Add directory creation for copilot engine --add-dir paths

  The copilot engine should review the --add-dir folders used in the args list and make sure those fol...

---

Cluster 2: Documentation

Size: 375 tasks (19.8% of total)

Performance:

• Success Rate: 77.1% (289 merged)
• Closed (not merged): 74
• Still open: 12

Characteristics:

• Avg Files Changed: 10.8
• Avg Additions: 485 lines
• Avg Comments: 1.1
• Avg Reviews: 1.5

Top Keywords: aw, gh, gh aw, githubnext, githubnext gh, githubnext gh aw, comments

Example Tasks:

• #2209: Comment on issue [smoke-detector] 🔍 Smoke Test Investigation - GenAIScript Invalid Model Name (gpt-4.1) #2157 regarding recurrence failure
• #2282: Extract 22 YAML generation functions from compiler.go
• #2283: Extract extraction functions from compiler.go

---

Cluster 3: New Features - Agentic Workflows

Size: 247 tasks (13.1% of total)

Performance:

• Success Rate: 79.8% (197 merged) ⭐ Best performing cluster
• Closed (not merged): 40
• Still open: 10

Characteristics:

• Avg Files Changed: 11.5
• Avg Additions: 1,848 lines
• Avg Comments: 2.2
• Avg Reviews: 1.7

Top Keywords: agentic, agentic workflow, workflow, workflows, update, shared, create

Example Tasks:

• #2100: Spread scheduled agentic workflows across 24 hours
• #2103: Add smoke-outpost workflow for investigating failed smoke test runs
• #2109: Add semantic function refactoring workflow for Go code analysis

---

Cluster 4: Documentation & CLI

Size: 203 tasks (10.7% of total)

Performance:

• Success Rate: 73.9% (150 merged)
• Closed (not merged): 46
• Still open: 7

Characteristics:

• Avg Files Changed: 20.1
• Avg Additions: 2,098 lines
• Avg Comments: 2.0
• Avg Reviews: 1.1

Top Keywords: cli, version, comments, issue_title, issue, section, issue_description

Example Tasks:

• #2156: Integrate gh-aw-firewall for Copilot engine
• #2208: Optimize CLI version checker workflow
• #2224: Change max-turns to 75 in cli-version-checker

---

Cluster 5: CI/CD & Workflows

Size: 114 tasks (6.0% of total)

Performance:

• Success Rate: 65.8% (75 merged) ⚠️ Lowest success rate
• Closed (not merged): 34
• Still open: 5

Characteristics:

• Avg Files Changed: 28.1 (highest complexity)
• Avg Additions: 2,949 lines
• Avg Comments: 4.6 (highest iteration count)
• Avg Reviews: 1.2

Top Keywords: mcp, server, mcp server, safe, tool, github, json

Example Tasks:

• #2167: Fix OpenCode MCP server integration
• #2219: Add tip about enabling agentic-workflows tool
• #2255: Replace GITHUB_PERSONAL_ACCESS_TOKEN with GITHUB_MCP_SERVER_TOKEN

---

Cluster 6: Testing & Code Quality

Size: 85 tasks (4.5% of total)

Performance:

• Success Rate: 78.8% (67 merged)
• Closed (not merged): 18
• Still open: 0

Characteristics:

• Avg Files Changed: 20.2
• Avg Additions: 1,988 lines
• Avg Comments: 1.3
• Avg Reviews: 1.2

Top Keywords: code, duplicate, duplicate code, analysis, tests, fix, commit

Example Tasks:

• #2171: Refactor duplicate MCP code patterns
• #2249: Refactor: Extract duplicate GitHub MCP remote config
• #2386: Optimize regex compilation (17-22x performance improvement)

---

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files	Top Keywords
1	New Features	868	72.7%	21.9	add, update, agent
2	Documentation	375	77.1%	10.8	aw, gh, gh aw
3	New Features - Agentic	247	79.8% ⭐	11.5	agentic, workflow
4	Documentation & CLI	203	73.9%	20.1	cli, version, comments
5	CI/CD & Workflows	114	65.8% ⚠️	28.1	mcp, server
6	Testing & Code Quality	85	78.8%	20.2	code, duplicate, tests

---

Key Insights

1. Agentic Workflow Tasks Perform Best

Tasks focused on creating or modifying agentic workflows (Cluster 3) have the highest success rate at 79.8%. These tasks typically:

• Have clear, specific objectives
• Modify existing patterns (workflow files)
• Average fewer files changed (11.5 vs 18.4 overall)
• Have well-defined constraints and examples

2. CI/CD & MCP Integration Tasks Are Most Complex

Tasks involving CI/CD and MCP server integration (Cluster 5) show:

• Lowest success rate: 65.8%
• Highest complexity: 28.1 files changed on average
• Most iterations: 4.6 comments per PR
• These tasks often involve configuration, permissions, and cross-system integration

3. Complexity Correlates with Lower Success

Analysis shows an inverse correlation between task complexity and success rate:

• Simple tasks (10-15 files): 75-80% success
• Medium tasks (15-25 files): 70-75% success
• Complex tasks (25+ files): 65-70% success

4. Documentation Tasks Have Moderate Success

Documentation-focused tasks (Clusters 2 & 4) show 73.9-77.1% success rates:

• Lower complexity than feature work
• More review iterations (human oversight)
• Clear success criteria

---

Recommendations

Based on this clustering analysis, we recommend:

1. Focus on High-Success Patterns

The 'New Features - Agentic Workflows' cluster shows 79.8% success rate. When creating new tasks:

• Use clear, specific prompts focused on single concerns
• Reference existing patterns and examples
• Provide explicit acceptance criteria
• Keep scope focused (aim for <15 files changed)

2. Break Down Complex CI/CD Tasks

CI/CD & MCP integration tasks (65.8% success) should be split into smaller, focused subtasks:

• Separate configuration changes from implementation
• Test integrations incrementally
• Provide more context about system interactions
• Consider multiple PRs for complex integrations

3. Manage Task Complexity

Clusters with high file changes show more iterations and lower success:

• Target: Keep tasks under 20 files when possible
• Strategy: Break large refactors into logical phases
• Benefit: Reduces cognitive load and failure points

4. Leverage Successful Prompt Patterns

The most successful clusters share common characteristics:

• ✅ Clear, single-purpose objectives
• ✅ Reference to existing code/patterns
• ✅ Specific file/directory context
• ✅ Explicit success criteria
• ❌ Avoid combining unrelated changes
• ❌ Avoid vague or open-ended requests

5. Consider Task Type When Setting Expectations

Different task types have different baseline success rates:

• Agentic Workflows: Expect 75-80% success
• Documentation: Expect 73-77% success
• General Features: Expect 70-73% success
• CI/CD Integration: Expect 65-70% success (needs more support)

---

Methodology

Data Collection:

• Source: 1,892 copilot agent PRs from last 30 days
• Extraction: Original prompts from PR bodies
• Enrichment: PR metadata (state, files, reviews, comments)

Analysis Technique:

• NLP Processing: TF-IDF vectorization with n-grams (1-3)
• Clustering: K-means with optimal k=6 (elbow method)
• Features: 150 top terms, min document frequency 3
• Validation: Silhouette score analysis

Cluster Themes: Inferred from top keywords and manual review of sample tasks

---

Analysis Tools: Python (scikit-learn, pandas), TF-IDF vectorization, K-means clustering
Date: 2025-12-12
Data Period: Last 30 days (1,892 tasks analyzed)

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6369.
Fair winds, matey! 🏴‍☠️