[prompt-clustering] 🔬 Copilot Agent Prompt Clustering Analysis - November 21, 2025 #4507
Closed
Replies: 1 comment
-
|
This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
🔬 Copilot Agent Prompt Clustering Analysis - 2025-11-21
Daily NLP-based clustering analysis of copilot agent task prompts using TF-IDF vectorization and K-means clustering.
Summary
Analysis Period: Last 30 days of copilot-created PRs
Total Tasks Analyzed: 984
Clusters Identified: 5
Overall Success Rate: 77.2%
Average Duration: 2.0 hours
Average Commits per Task: 3.6
Average Files Changed: 14.1
This analysis examined 984 copilot-generated pull requests, extracting task prompts from PR bodies and applying natural language processing techniques to identify common patterns and themes. The clustering algorithm identified 5 distinct task categories, revealing insights into how copilot agents are being used and which types of tasks perform best.
Full Analysis Report
General Insights
Most Common Task Type: Updates & Changes accounts for 38.2% of all tasks, indicating that copilot agents are primarily used for updates & changes work.
Highest Success Rate: Bug Fixes tasks have the highest success rate at 78.9%, suggesting these types of tasks are well-suited for agent automation.
Task Complexity Patterns: Updates & Changes tasks are the most complex (avg 3.9 commits, 18.4 files), while General Tasks tasks are simpler (avg 3.6 commits, 8.4 files).
Success Rate Variance: There is a 5.5% difference between the best and worst performing task types, indicating some task types are more suitable for agent automation than others.
Execution Speed: Updates & Changes tasks complete fastest (avg 1.5 hours), while New Features tasks take longer (avg 3.3 hours).
Cluster Analysis
Cluster 4: Updates & Changes
Size: 376 tasks (38.2% of total)
Success Rate: 76.1%
Average Metrics:
Top Keywords:
update, github, md, workflow, file, error, files, codeCharacteristics: This cluster represents tasks focused on updating existing functionality, modifying workflows, and making incremental changes to documentation and configuration files.
Example Tasks:
Cluster 1: Bug Fixes
Size: 350 tasks (35.6% of total)
Success Rate: 78.9%
Average Metrics:
Top Keywords:
section details, issue resolve, details original, original issue, resolve, original, details, sectionCharacteristics: Tasks in this cluster are primarily focused on resolving issues, fixing bugs, and addressing specific problems reported in the repository.
Example Tasks:
Cluster 2: General Tasks
Size: 128 tasks (13.0% of total)
Success Rate: 78.1%
Average Metrics:
Top Keywords:
agentic, workflow, agentic workflow, workflows, agentic workflows, update, agent, createCharacteristics: General development tasks that don't fit neatly into other categories, often involving workflow management and configuration.
Example Tasks:
Cluster 3: New Features
Size: 100 tasks (10.2% of total)
Success Rate: 76.0%
Average Metrics:
Top Keywords:
add, command, firewall, compile, update, workflows, logs, usingCharacteristics: Tasks that implement new functionality, add new commands, or introduce new capabilities to the system.
Example Tasks:
Cluster 5: Documentation Updates
Size: 30 tasks (3.0% of total)
Success Rate: 73.3%
Average Metrics:
Top Keywords:
pull, pull request, request, comment, create, workflow, safe, updateCharacteristics: This cluster contains tasks related to creating, updating, and improving documentation, comments, and pull request descriptions.
Example Tasks:
Success Rate by Cluster
Sample Data Table
Showing representative sample of 984 total PRs
Key Findings
Most Common Task Type: Updates & Changes accounts for 38.2% of all tasks, indicating that copilot agents are primarily used for updates & changes work.
Highest Success Rate: Bug Fixes tasks have the highest success rate at 78.9%, suggesting these types of tasks are well-suited for agent automation.
Task Complexity Patterns: Updates & Changes tasks are the most complex (avg 3.9 commits, 18.4 files), while General Tasks tasks are simpler (avg 3.6 commits, 8.4 files).
Success Rate Variance: There is a 5.5% difference between the best and worst performing task types, indicating some task types are more suitable for agent automation than others.
Execution Speed: Updates & Changes tasks complete fastest (avg 1.5 hours), while New Features tasks take longer (avg 3.3 hours).
Recommendations
Optimize for High-Success Task Types: Focus agent usage on Bug Fixes tasks which show 78.9% success rate. These tasks are well-defined and benefit most from automation.
Improve Prompts for Low-Success Tasks: Documentation Updates tasks have the lowest success rate (73.3%). Review failed PRs in this cluster to identify common failure patterns and improve prompt engineering.
Task Decomposition for Complex Work: Tasks in the Documentation Updates cluster average 4.1 commits. Consider breaking down complex tasks into smaller, more manageable subtasks for better results.
Standardize Prompt Patterns: Analysis reveals 5 distinct task patterns. Create template prompts for each cluster type to ensure consistency and improve success rates across all task categories.
Monitor Task Duration: New Features tasks take 3.3 hours on average. Set appropriate timeout expectations and consider whether these tasks need additional context or guidance.
Methodology: TF-IDF vectorization with 100 features, K-means clustering with 5 clusters, PCA for visualization
Data Files: Full analysis data and visualizations available in workflow artifacts
Generated by Copilot Agent Prompt Clustering Analysis
Beta Was this translation helpful? Give feedback.
All reactions