🎯 Repository Quality Improvement Report - CLI Feedback Quality #4666
Closed
Replies: 1 comment
-
|
This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
🎯 Repository Quality Improvement Report - CLI Feedback Quality
Analysis Date: 2024-11-24
Focus Area: CLI Feedback Quality
Strategy Type: Custom
Custom Area: Yes - This focus area addresses the specific challenge of providing clear, actionable feedback to users (both human developers and AI agents) when working with agentic workflows. Given that gh-aw is a tool for creating AI-driven workflows, exceptional error messages and user guidance are critical for adoption and debugging.
Executive Summary
The gh-aw CLI demonstrates strong adoption of structured console formatting (87% formatted output) and maintains consistent messaging patterns across 754 formatted output locations. However, analysis reveals significant opportunities for improvement in error message helpfulness and command documentation. Only 7.1% of error messages (44 out of 613) provide actionable suggestions or examples, leaving 92.9% of errors as basic technical messages that may confuse users. Additionally, critical commands like
compile,list,run, andstatuslack Short/Long descriptions and examples in their help text, creating gaps in user guidance.The impact is particularly acute for this tool because users are learning to work with agentic workflows—a novel paradigm. When compilation fails or workflows behave unexpectedly, users need clear guidance on what went wrong and how to fix it. Improving CLI feedback quality will reduce support burden, accelerate user onboarding, and improve the overall developer experience.
Full Analysis Report
Focus Area: CLI Feedback Quality
Current State Assessment
The gh-aw CLI has established good foundations for user feedback through the
pkg/consolepackage, which provides formatting utilities for success, error, warning, info, and progress messages. The codebase shows strong discipline in using these utilities, with 87% of output properly formatted.Metrics Collected:
Findings
Strengths
console.FormatErrorWithSuggestions()function demonstrates awareness of good error message designAreas for Improvement
Low Helpful Error Rate (7.1%):
Incomplete Command Help Text:
compilecommand: No Short/Long description or exampleslist,run,status,removecommands: Missing help textaddcommand has comprehensive help with 3 examplesGeneric Error Messages:
fmt.Errorf("failed to X: %w", err)provides technical error but no context"failed to create .github/workflows directory: %w"doesn't explain why or how to fixInconsistent Example Provision:
Detailed Analysis
Error Message Categories
Category 1: Basic Technical Errors (92.9%)
Category 2: Helpful Errors with Suggestions (7.1%)
Command Help Quality Analysis
The
compilecommand is particularly concerning as it's one of the most frequently used commands and currently has no help text.User Impact Scenarios
Scenario 1: New User Compiling First Workflow
gh aw compile my-workflow.md"failed to read file: no such file or directory"Scenario 2: Workflow Compilation Failure
"generated lock file is not valid YAML: %w"Scenario 3: Security Scanner Failure
"zizmor security scan failed: %w"--verbosefor details🤖 Tasks for Copilot Agent
NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.
Improvement Tasks
The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.
Task 1: Add Comprehensive Help Text to Core Commands
Priority: High
Estimated Effort: Medium
Focus Area: CLI Feedback Quality
Description:
Add Short descriptions, Long descriptions, and usage Examples to commands that currently lack them:
compile,list,run,status, andremove. These are core commands that users frequently use, and they need clear guidance on usage patterns.Acceptance Criteria:
addcommandCode Region:
pkg/cli/*_command.go(compile_command.go, list_command.go, run_command.go, status_command.go, remove_command.go)Reference the
addcommand (add_command.go) for the quality standard to match.Focus on errors in:
✓ Compilation completed in 12.3s
Summary:
Total workflows: 12
Successfully compiled: 10
Failed: 2
Skipped (up-to-date): 0
Task 4: Create Error Message Style Guide Documentation
Priority: Medium
Estimated Effort: Small
Focus Area: CLI Feedback Quality
Description:
Document the error message patterns and best practices for contributors. Create a style guide that explains when to use different console formatting functions, how to structure helpful error messages, and provides before/after examples. This will help maintain consistency as the codebase grows and new contributors add features.
Acceptance Criteria:
.github/instructions/error-message-style.instructions.mdCode Region:
.github/instructions/(new file)2. File/Path Errors
Include expected path, current working directory context, and navigation help.
Pattern: What's missing + Where we looked + How to fix
3. Tool/Dependency Errors
Explain what tool is missing, why it's needed, and how to install it.
Pattern: Tool name + Purpose + Installation
4. Validation Errors
Include specific location of problem, explanation, and how to fix.
Pattern: Specific issue + Location + Fix
5. Operation Failures
Wrap technical errors with user-friendly context and next steps.
Pattern: What failed + Why + Debug options
Console Formatting Functions
FormatErrorMessage(msg string)
Use for simple error output without suggestions.
FormatErrorWithSuggestions(msg string, suggestions []string)
Use this for most user-facing errors. Provides actionable next steps.
FormatWarningMessage(msg string)
Use for non-critical issues that don't stop execution.
FormatInfoMessage(msg string)
Use for informational output during normal operation.
FormatSuccessMessage(msg string)
Use for completion confirmations.
FormatProgressMessage(msg string)
Use for progress updates during long operations.
Before/After Examples
Example 1: File Operations
Before:
return fmt.Errorf("failed to create directory: %w", err)After:
Example 2: Invalid Flags
Before:
return fmt.Errorf("--purge flag can only be used when compiling all markdown files")After:
Example 3: Tool Execution
Before:
return fmt.Errorf("zizmor security scan failed: %w", err)After:
Anti-patterns to Avoid
❌ Generic wrapped errors without context
❌ Technical jargon without explanation
❌ Errors without actionable next steps
❌ Assuming user knowledge
Good Patterns to Follow
✅ Specific, contextual, actionable
✅ Include examples of correct usage
✅ Link to documentation when relevant
✅ Suggest debug flags for complex issues
✅ Use consistent terminology from user documentation
Testing Error Messages
When adding new error messages, test them by:
References
Good examples in the codebase:
pkg/cli/commands.go- workflow resolution errorspkg/cli/resolver.go- file not found with suggestionspkg/workflow/resolve.go- import resolution errorsAdd similar documentation for each showing:
Add a package-level documentation comment explaining:
Beta Was this translation helpful? Give feedback.
All reactions