[prompt-clustering] Copilot Agent Prompt Clustering Analysis - Dec 2025

[github-actions[bot]]

Daily NLP-based clustering analysis of copilot agent task prompts to identify patterns, success factors, and optimization opportunities.

Summary

Analysis Date: 2025-12-09
Analysis Period: Approximately 90 days (Nov-Dec 2025)
Total Tasks Analyzed: 1750
Clusters Identified: 10
Overall Success Rate: 74.9% (1310 merged, 440 closed)

Key Insights

• Most Common Task Type: Documentation tasks dominate the workload, representing 20.9% of all agent tasks with a 71.9% success rate.
• Highest Success Rate: Testing tasks (Cluster 7) achieve 83.3% success, demonstrating that well-structured testing prompts work exceptionally well.
• Needs Improvement: Build tasks (Cluster 6) show a 61.9% success rate, indicating opportunities for prompt engineering improvements in this area.
• Task Complexity: On average, tasks modify 19.4 files. Tasks in the optimal complexity range (2-5 files) tend to have higher success rates.

Full Clustering Analysis

Cluster Breakdown

The 10 identified clusters represent distinct categories of agent tasks based on NLP analysis of task prompts. Each cluster shares similar keywords, themes, and characteristics.

Cluster 2: Documentation

Size: 366 tasks (20.9% of total)
Success Rate: 71.9% (263 merged, 103 closed)
Avg Files Changed: 18.7
Avg Commits: 3.7
Avg Lines Added: 1059

Top Keywords: agent, add, firewall, github, file, logs, javascript

Characteristics: Tasks in this cluster are primarily focused on documentation-related work. typically involving multiple file changes. representing a major category of agent tasks.

Example PRs:

• #3147: Delete smoke-opencode workflow
• #4976: Add security documentation guides

Cluster 3: Documentation

Size: 356 tasks (20.3% of total)
Success Rate: 75.3% (268 merged, 88 closed)
Avg Files Changed: 19.3
Avg Commits: 3.7
Avg Lines Added: 810

Top Keywords: update, job, workflows, remove, make, comment, workflow

Characteristics: Tasks in this cluster are primarily focused on documentation-related work. This cluster shows good success rates. typically involving multiple file changes. representing a major category of agent tasks.

Example PRs:

• #2189: [WIP] Recompile workflows for updated processes
• #5248: Revert AWF to v0.4.0

Cluster 10: Build

Size: 337 tasks (19.3% of total)
Success Rate: 79.5% (268 merged, 69 closed)
Avg Files Changed: 10.3
Avg Commits: 3.2
Avg Lines Added: 475

Top Keywords: gh aw, githubnext gh aw, githubnext gh, aw, githubnext, gh, comments

Characteristics: Tasks in this cluster are primarily focused on build-related work. This cluster shows good success rates. typically involving multiple file changes. representing a major category of agent tasks.

Example PRs:

• #4776: Add --ref flag to gh aw status and rename --branch to --ref in gh aw logs
• #3259: Add write permissions for githubnext/gh-aw in devcontainer.json

Cluster 9: Documentation

Size: 180 tasks (10.3% of total)
Success Rate: 78.9% (142 merged, 38 closed)
Avg Files Changed: 9.6
Avg Commits: 3.7
Avg Lines Added: 1686

Top Keywords: workflow, agentic, agentic workflow, shared, create, update, add

Characteristics: Tasks in this cluster are primarily focused on documentation-related work. This cluster shows good success rates. typically involving multiple file changes.

Example PRs:

• #3148: Remove smoke-copilot.firewall workflow
• #3442: Add super-linter agentic workflow for automated code quality analysis

Cluster 5: Bugfix

Size: 109 tasks (6.2% of total)
Success Rate: 66.1% (72 merged, 37 closed)
Avg Files Changed: 13.8
Avg Commits: 3.4
Avg Lines Added: 836

Top Keywords: comments, issue, section, issue_title, cli, issue_description, original

Characteristics: Tasks in this cluster are primarily focused on bugfix-related work. typically involving multiple file changes.

Example PRs:

• #3726: Update golang.org/x/text from v0.30.0 to v0.31.0
• #4363: Refactor: Extract duplicate log parser markdown generation to shared module

Cluster 8: Feature

Size: 104 tasks (5.9% of total)
Success Rate: 71.2% (74 merged, 30 closed)
Avg Files Changed: 34.5
Avg Commits: 4.6
Avg Lines Added: 2214

Top Keywords: safe, safe output, output, add, outputs, create, issue

Characteristics: Tasks in this cluster are primarily focused on feature-related work. typically involving multiple file changes.

Example PRs:

• #5340: Add assign-to-user safe output type
• #5533: Add safe-inputs documentation to site navigation

Cluster 6: Build

Size: 97 tasks (5.5% of total)
Success Rate: 61.9% (60 merged, 37 closed)
Avg Files Changed: 28.7
Avg Commits: 4.6
Avg Lines Added: 3146

Top Keywords: mcp, server, mcp server, tool, github, json, safe

Characteristics: Tasks in this cluster are primarily focused on build-related work. This cluster has a lower success rate and may benefit from improved prompts. typically involving multiple file changes.

Example PRs:

• #4725: Upgrade @sentry/mcp-server to 0.23.1
• #5595: [WIP] Upgrade @sentry/mcp-server to version 0.24.0

Cluster 4: Documentation

Size: 90 tasks (5.1% of total)
Success Rate: 80.0% (72 merged, 18 closed)
Avg Files Changed: 24.8
Avg Commits: 2.9
Avg Lines Added: 234

Top Keywords: version, cli, changes, package, release, update, updates

Characteristics: Tasks in this cluster are primarily focused on documentation-related work. This cluster shows good success rates. typically involving multiple file changes.

Example PRs:

• #4708: Revert copilot cli to 0.0.358
• #3947: [WIP] Revert changeset to version in commit 9e0ba6b

Cluster 1: Refactoring

Size: 57 tasks (3.3% of total)
Success Rate: 80.7% (46 merged, 11 closed)
Avg Files Changed: 20.0
Avg Commits: 3.4
Avg Lines Added: 2274

Top Keywords: code, duplicate code, duplicate, analysis, refactoring, duplication, helper

Characteristics: Tasks in this cluster are primarily focused on refactoring-related work. This cluster has an excellent success rate. typically involving multiple file changes. representing a specialized subset of tasks.

Example PRs:

• #2386: Optimize regex compilation in hot paths - 17-22x performance improvement
• #3045: Enable duplicate code detection for .cjs files

Cluster 7: Testing

Size: 54 tasks (3.1% of total)
Success Rate: 83.3% (45 merged, 9 closed)
Avg Files Changed: 14.5
Avg Commits: 3.3
Avg Lines Added: 606

Top Keywords: fix, tests, javascript, issues, workflows, test, error

Characteristics: Tasks in this cluster are primarily focused on testing-related work. This cluster has an excellent success rate. typically involving multiple file changes. representing a specialized subset of tasks.

Example PRs:

• #4337: Remove tracked test artifacts and ignore *.tmp files
• #4365: Fix test isolation in collect_ndjson_output.test.cjs

Success Rate Comparison

Rank	Cluster	Theme	Tasks	Success Rate	Merged	Closed
1	7	Testing	54	83.3%	45	9
2	1	Refactoring	57	80.7%	46	11
3	4	Documentation	90	80.0%	72	18
4	10	Build	337	79.5%	268	69
5	9	Documentation	180	78.9%	142	38
6	3	Documentation	356	75.3%	268	88
7	2	Documentation	366	71.9%	263	103
8	8	Feature	104	71.2%	74	30
9	5	Bugfix	109	66.1%	72	37
10	6	Build	97	61.9%	60	37

Recommendations

Based on the clustering analysis, here are actionable recommendations to improve agent task success rates:

1. Leverage High-Success Patterns: The 'Testing' cluster (Cluster 7) achieves 83.3% success. Analyze prompts in this cluster to identify effective prompt patterns and apply them to other task types.
2. Improve Low-Success Areas: The 'Build' cluster (Cluster 6) has only 61.9% success rate. Review failed tasks in this cluster to identify common failure patterns. Consider providing more context, clearer requirements, or breaking down complex tasks.
3. Optimize Common Tasks: The 'Documentation' cluster (Cluster 2) represents 20.9% of all tasks. Creating specialized prompt templates or guidelines for this task type could significantly improve overall efficiency.
4. Prompt Clarity: Tasks with clear, specific prompts (50-150 words) tend to have higher success rates. Avoid overly brief prompts that lack context and overly verbose prompts that may confuse the agent.
5. Task Scoping: Tasks that involve 2-5 file changes have the highest success rates. Consider breaking down larger refactoring tasks into smaller, focused subtasks.

Methodology

Data Collection: Analyzed 1,750 copilot agent PRs from the githubnext/gh-aw repository.

Text Processing:

• Extracted task prompts from PR bodies (specifically from 'Original prompt' sections)
• Cleaned and normalized text (removed markdown, URLs, special characters)
• Filtered out prompts shorter than 20 characters

Clustering Algorithm:

• TF-IDF vectorization with 200 features, 1-3 word n-grams
• K-means clustering with optimal k determined by silhouette score
• Identified 10 distinct clusters

Metrics Analyzed:

• Success rate (merged vs closed PRs)
• Task complexity (files changed, lines added/deleted)
• Review engagement (number of reviews, comments)
• Commit patterns (number of commits per task)

---

Analysis Run: §20075254757

AI generated by Copilot Agent Prompt Clustering Analysis

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Copilot Agent Prompt Clustering Analysis.
🗺️ A newer treasure map awaits ye at Discussion #6059.
Fair winds, matey! 🏴‍☠️