🎯 Repository Quality Improvement Report - Workflow Validation and Error Feedback Quality

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Validation and Error Feedback Quality

Analysis Date: 2025-11-14
Focus Area: Workflow Validation and Error Feedback Quality
Strategy Type: Custom
Custom Area: Yes - This focus area examines repository-specific concerns around how the gh-aw compiler validates agentic workflows and provides actionable feedback to users when errors occur.

Executive Summary

The gh-aw repository demonstrates a mature and well-architected validation system with 16 dedicated validation implementation files, 510+ validation references, and comprehensive JSON schema validation with strict additionalProperties: false enforcement. The validation architecture is thoughtfully organized by domain (strict mode, permissions, expressions, packages, etc.) with clear separation of concerns.

However, error message quality and actionability present opportunities for improvement. While the codebase has 257 error generation points and includes helpful error formatting utilities (console.FormatErrorMessage, frontmatter_error.go), many error messages lack actionable suggestions, examples, or "did you mean" guidance. The time_delta.go file demonstrates excellent error message patterns with specific format examples, but this pattern isn't consistently applied across the codebase.

Key Findings:

• ✅ Excellent: 16 validation files with domain-driven architecture
• ✅ Excellent: JSON schema validation with strict property enforcement (120 schema references)
• ⚠️ Good but improvable: Error formatting infrastructure exists but underutilized
• ❌ Needs attention: Many error messages lack actionable suggestions (only ~24 use console formatting)
• ❌ Needs attention: Limited "did you mean" or suggestion-based error messages
• ⚠️ Documentation gap: Error messages and troubleshooting guidance could be more prominent in docs

Full Analysis Report

Focus Area: Workflow Validation and Error Feedback Quality

Current State Assessment

Metrics Collected:

Metric	Value	Status
Validation implementation files	16 files	✅ Excellent
Validation references in codebase	510+	✅ Excellent
Error generation points	257	✅ Comprehensive
Console-formatted error messages	~24	⚠️ Underutilized
JSON schema size	3,564 lines (main)	✅ Comprehensive
Schema additionalProperties: false	120 instances	✅ Strict validation
Test files covering validation/errors	139	✅ Well-tested
Error-focused documentation files	~10	⚠️ Could expand

Findings

Strengths

1. World-class validation architecture - The domain-driven validation structure (validation.go header documentation, separate files for strict_mode_validation.go, pip_validation.go, npm_validation.go, etc.) demonstrates excellent software design
2. Comprehensive schema validation - JSON schemas with strict additionalProperties: false catch typos and invalid fields before they cause runtime issues
3. Strong error infrastructure - The console package and frontmatter_error.go provide excellent error formatting capabilities with file/line/column positioning
4. Excellent test coverage - 139 test files cover validation and error scenarios, including edge cases
5. Security-first validation - Strict mode validation (strict_mode_validation.go) enforces security constraints for production workflows
6. Good error examples exist - time_delta.go demonstrates excellent error messages with format examples and specific guidance

Areas for Improvement

1. Inconsistent error message quality - While time_delta.go provides examples like "Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", many other error messages lack similar guidance
2. Missing "did you mean" suggestions - No evidence of fuzzy matching or suggestion-based error recovery (e.g., when user types permisions instead of permissions)
3. Underutilized console formatting - Only ~24 instances of console.FormatErrorMessage despite 257 error generation points
4. Limited actionable hints - Many errors state what's wrong but not how to fix it (e.g., "strict mode: write permission 'contents: write' is not allowed" could suggest using safe-outputs instead)
5. Documentation gap - Error messages and troubleshooting guidance scattered across docs; could benefit from centralized error reference
6. Validation error test coverage - Only 30 tests explicitly validate error message content/format

Detailed Analysis

Validation System Architecture

The validation system is exemplary in its organization:

pkg/workflow/
├── validation.go                    # Package documentation
├── strict_mode_validation.go        # Security constraints
├── repository_features_validation.go # Capability detection
├── schema_validation.go             # GitHub Actions schema
├── runtime_validation.go            # Packages, containers, expressions
├── agent_validation.go              # Agent files
├── pip_validation.go                # Python packages
├── npm_validation.go                # NPM packages
├── docker_validation.go             # Container images
├── expression_validation.go         # Expression safety
├── engine_validation.go             # AI engine config
├── mcp_config_validation.go         # MCP servers
├── template_validation.go           # Templates
├── bundler_validation.go            # JavaScript bundler
├── permissions_validator.go         # Permissions
├── step_order_validation.go         # Step ordering
└── frontmatter_error.go             # Error formatting

This structure follows the Single Responsibility Principle and makes the codebase highly maintainable.

Error Message Quality Analysis

Excellent Error Messages (time_delta.go):

fmt.Errorf("invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m", deltaStr)
fmt.Errorf("minute unit 'm' is not allowed for stop-after. Minimum unit is hours 'h'. Use +%dh instead of +%dm", (value+59)/60, value)

These messages are excellent because they:

• State what's wrong
• Provide examples of valid formats
• Suggest specific fixes with actual values

Typical Error Messages (strict_mode_validation.go):

fmt.Errorf("strict mode: write permission '%s: write' is not allowed", scope)
fmt.Errorf("strict mode: 'network' configuration is required")

These messages are clear but could be improved with:

• Why this restriction exists
• How to achieve the desired outcome (e.g., "use safe-outputs.create-issue instead")
• Link to documentation

Generic Error Messages:

return "", fmt.Errorf("manual-approval value must be a string")
return "", fmt.Errorf("invalid on: section format")

These messages need:

• What was received vs. what was expected
• Examples of valid values
• Specific fix suggestions

Schema Validation Effectiveness

The JSON schemas are comprehensive (3,564 lines for main schema) and use strict validation with additionalProperties: false (120 instances). This catches typos like permisions, engnie, toolz early in the compilation process.

However, the schema validation errors could be enhanced with:

• Fuzzy matching to suggest "did you mean 'permissions'?" when user types permisions
• Enumeration of valid field names when invalid field detected
• Links to schema documentation for complex fields

Error Formatting Infrastructure

The console package provides excellent error formatting:

• File/line/column positioning
• Context lines around errors
• Syntax highlighting potential
• Structured error output

But only ~24 uses of console.FormatErrorMessage suggests this infrastructure is underutilized. Many errors use plain fmt.Errorf instead.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for processing.

Improvement Tasks

Task 1: Implement "Did You Mean" Suggestions for Schema Validation

Priority: High
Estimated Effort: Medium
Focus Area: Workflow Validation and Error Feedback Quality

Description:
When users make typos in workflow frontmatter fields (e.g., permisions instead of permissions, engnie instead of engine, toolz instead of tools), the compiler should suggest the correct field name using fuzzy string matching.

Implement Levenshtein distance-based suggestion system that:

1. Detects invalid field names from JSON schema validation errors
2. Compares invalid field to all valid schema fields
3. Suggests closest match if edit distance ≤ 3
4. Formats error with "did you mean 'permissions'?" suggestion

Acceptance Criteria:

• Detects typos with edit distance ≤ 3 from valid field names
• Suggests single best match when similarity is high (distance ≤ 2)
• Lists multiple suggestions when multiple fields are similar (distance = 3)
• Integrates with existing console.FormatErrorMessage infrastructure
• Includes unit tests for common typos: permisions, engnie, toolz, timeout_minute, runs_on
• Handles both top-level frontmatter fields and nested fields (e.g., engine.model)

Code Region: pkg/workflow/schema_validation.go, pkg/parser/

Create a new file `pkg/workflow/schema_fuzzy_match.go` with:

1. Function `suggestFieldName(invalidField string, validFields []string) []string`
   - Uses Levenshtein distance algorithm
   - Returns suggestions with distance ≤ 3, sorted by distance
   
2. Function `enhanceSchemaValidationError(err error, schema map[string]interface{}) error`
   - Parses schema validation error for invalid field names
   - Calls `suggestFieldName` with schema's valid properties
   - Formats enhanced error using `console.FormatErrorMessage`
   
3. Integration point in `schema_validation.go`:
   - Wrap schema validation errors with `enhanceSchemaValidationError`
   - Extract valid field names from JSON schema before validation
   
4. Tests in `pkg/workflow/schema_fuzzy_match_test.go`:
   - Test common typos return correct suggestions
   - Test exact matches don't suggest
   - Test very different strings don't suggest
   - Test multiple similar suggestions

---

Task 2: Add Actionable Hints to Strict Mode Validation Errors

Priority: High
Estimated Effort: Small
Focus Area: Workflow Validation and Error Feedback Quality

Description:
Enhance strict mode validation error messages to include actionable hints about alternative approaches. When strict mode rejects an operation, users should understand why and how to achieve their goal safely.

Update all strict mode error messages in pkg/workflow/strict_mode_validation.go to:

1. Explain WHY the restriction exists (security concern)
2. Suggest HOW to achieve the desired outcome using safe alternatives
3. Link to relevant documentation

Acceptance Criteria:

• All 4+ strict mode error messages include actionable hints
• Error messages explain security rationale
• Error messages suggest safe-outputs alternatives where applicable
• Error messages use console.FormatErrorMessage with hint field
• Documentation links are valid and helpful
• Unit tests verify hint content is present

Code Region: pkg/workflow/strict_mode_validation.go

Update error messages in `strict_mode_validation.go`:

1. `validateStrictPermissions` error:
   - Current: "strict mode: write permission '%s: write' is not allowed"
   - Enhanced: "strict mode: write permission '%s: write' is not allowed for security reasons. Use 'safe-outputs.create-issue' or 'safe-outputs.create-pull-request' to perform write operations safely. See: (redacted)"
   
2. `validateStrictNetwork` error:
   - Current: "strict mode: 'network' configuration is required"
   - Enhanced: "strict mode: 'network' configuration is required to prevent unrestricted network access. Add 'network: { allowed: [...] }' or 'network: defaults' to your frontmatter. See: (redacted)"
   
3. Wildcard error:
   - Current: "strict mode: wildcard '*' is not allowed in network.allowed domains"
   - Enhanced: "strict mode: wildcard '*' is not allowed in network.allowed domains to prevent unrestricted internet access. Specify explicit domains or use ecosystem identifiers like 'python', 'node', 'containers'. See: (redacted)"

4. Use `console.CompilerError` struct with `Hint` field for better formatting

5. Add tests to verify hint text is included in error output

---

Task 3: Standardize Error Message Format with Examples

Priority: Medium
Estimated Effort: Medium
Focus Area: Workflow Validation and Error Feedback Quality

Description:
Create and enforce a standard format for validation error messages across the codebase, following the excellent pattern established in time_delta.go. All validation errors should include: (1) what's wrong, (2) what was expected/valid format, (3) example of correct usage when applicable.

Acceptance Criteria:

• Create error message style guide in .github/instructions/error-messages.instructions.md
• Identify 20+ error messages that lack examples/guidance
• Refactor error messages to include format examples
• Add validation tests that check error message content
• Update at least 5 high-impact error messages (compile failures, schema violations, permission errors)
• Document before/after examples in commit message

Code Region: pkg/workflow/*.go (validation files), .github/instructions/

1. Create `.github/instructions/error-messages.instructions.md`:

```markdown
# Error Message Style Guide

All validation error messages should follow this format:

**Template:**
"[what's wrong]. [what's expected]. [example of correct usage]"

**Good Examples:**
- "invalid time delta format: +%s. Expected format like +25h, +3d, +1w, +1mo, +1d12h30m"
- "manual-approval value must be a string, got %T. Example: manual-approval: \"production\""
- "unknown field 'permisions' in frontmatter. Valid fields include: permissions, engine, tools, safe-outputs"

**Bad Examples:**
- "invalid format" (lacks specifics)
- "manual-approval value must be a string" (no example)
- "validation failed" (too generic)

**When to include examples:**
- Always for format/syntax errors
- Always for enum/choice fields
- When expected value format is not obvious
- When there are multiple valid formats

2. Refactor high-impact error messages:

**(redacted) pkg/workflow/manual_approval.go

// Before:
return "", fmt.Errorf("manual-approval value must be a string")

// After:
return "", fmt.Errorf("manual-approval value must be a string, got %T. Example: manual-approval: \"production\"", val)

**(redacted) pkg/workflow/engine_validation.go

// Before:
return fmt.Errorf("invalid engine: %s", engineID)

// After:
return fmt.Errorf("invalid engine: %s. Valid engines are: copilot, claude, codex, custom. Example: engine: copilot", engineID)

3. Add tests validating error message content:

func TestErrorMessageFormat(t *testing.T) {
    err := validateSomething(invalidInput)
    if err == nil {
        t.Fatal("expected error")
    }
    
    // Error should explain what's wrong
    if !strings.Contains(err.Error(), "invalid") {
        t.Error("error should state what's wrong")
    }
    
    // Error should include example
    if !strings.Contains(err.Error(), "Example:") {
        t.Error("error should include example")
    }
}

4. Target these specific files for improvement:
   • pkg/workflow/manual_approval.go
   • pkg/workflow/engine_validation.go
   • pkg/workflow/mcp_config_validation.go
   • pkg/workflow/permissions_validator.go
   • pkg/workflow/docker_validation.go

---

#### Task 4: Create Centralized Error Reference Documentation

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Workflow Validation and Error Feedback Quality

**Description:**
Create a centralized error reference guide that documents common compilation errors, their causes, and solutions. This should be easily searchable and linked from error messages.

**Acceptance Criteria:**
- [ ] Create `docs/src/content/docs/troubleshooting/errors.md` with error reference
- [ ] Document at least 15 common error scenarios with solutions
- [ ] Include search-friendly error codes or error message prefixes
- [ ] Organize by category: Schema Validation, Permissions, Network, MCP, Engine, etc.
- [ ] Add "See also" links to related documentation
- [ ] Update main documentation navigation to include troubleshooting section
- [ ] Include examples of both incorrect and correct workflow configurations

**Code Region:** `docs/src/content/docs/troubleshooting/`

```markdown
Create `docs/src/content/docs/troubleshooting/errors.md`:

```markdown
---
title: Error Reference
description: Common compilation errors and their solutions
---

# Error Reference

This page documents common errors encountered when compiling agentic workflows and how to resolve them.

## Schema Validation Errors

### Unknown Field in Frontmatter

**Error Message:**

unknown field 'permisions' in frontmatter

**Cause:** Typo or unsupported field in workflow frontmatter YAML.

**Solution:** Check field spelling against the [Frontmatter Reference](/reference/frontmatter/). Common typos:
- `permisions` → `permissions`
- `engnie` → `engine`
- `toolz` → `tools`

**Example:**
```yaml
# ❌ Incorrect
---
permisions:
  contents: read
---

# ✅ Correct
---
permissions:
  contents: read
---

Invalid Engine Value

Error Message:

invalid engine: copliot. Valid engines are: copilot, claude, codex, custom

Cause: Misspelled or unsupported engine identifier.

Solution: Use one of the supported engines: copilot, claude, codex, or custom.

See also: Engine Configuration

---

Strict Mode Errors

Write Permission Not Allowed

Error Message:

strict mode: write permission 'contents: write' is not allowed

Cause: Strict mode enforces security by restricting write permissions.

Solution: Use safe-outputs instead:

# ❌ Direct write permission
---
strict: true
permissions:
  contents: write
---

# ✅ Use safe-outputs
---
strict: true
permissions:
  contents: read
safe-outputs:
  create-pull-request:
---

See also: Safe Outputs, Strict Mode

---

[Continue with 12+ more error scenarios...]

Update `docs/src/config.ts` to add troubleshooting to navigation:
```typescript
sidebar: [
  // ... existing items
  {
    label: 'Troubleshooting',
    items: [
      'troubleshooting/errors',
      'troubleshooting/faq',
    ],
  },
]

---

#### Task 5: Implement Validation Error Summary Report

**Priority**: Low  
**Estimated Effort**: Small  
**Focus Area**: Workflow Validation and Error Feedback Quality

**Description:**
When compilation fails with multiple validation errors, provide a summary report that groups errors by category and suggests a recommended order for fixing them. This helps users tackle the most critical issues first and understand the scope of problems.

**Acceptance Criteria:**
- [ ] Collect all validation errors during compilation (don't fail on first error)
- [ ] Group errors by category: Schema, Permissions, Network, Security, etc.
- [ ] Sort by severity: Critical → High → Medium → Low
- [ ] Display summary with error counts per category
- [ ] Show recommended fix order
- [ ] Link to error reference documentation for each category
- [ ] Include option to show/hide verbose error details

**Code Region:** `pkg/workflow/compiler.go`, `pkg/console/`

```markdown
Update `pkg/workflow/compiler.go` to collect multiple errors:

1. Add error collection during validation:
```go
type ValidationResults struct {
    Errors []ValidationError
    Warnings []ValidationWarning
}

type ValidationError struct {
    Category string // "schema", "permissions", "network", "security"
    Severity string // "critical", "high", "medium", "low"
    Message  string
    File     string
    Line     int
    Hint     string
}

func (c *Compiler) validateWorkflow(frontmatter map[string]any) (*ValidationResults, error) {
    results := &ValidationResults{}
    
    // Collect all errors instead of returning on first error
    if err := c.validateSchema(frontmatter); err != nil {
        results.Errors = append(results.Errors, ValidationError{
            Category: "schema",
            Severity: "high",
            Message:  err.Error(),
        })
    }
    
    if err := c.validatePermissions(frontmatter); err != nil {
        results.Errors = append(results.Errors, ValidationError{
            Category: "permissions",
            Severity: "critical",
            Message:  err.Error(),
        })
    }
    
    // ... more validations
    
    return results, nil
}

2. Add summary formatter in pkg/console/validation_summary.go:

func FormatValidationSummary(results *ValidationResults) string {
    // Group by category
    // Sort by severity
    // Format with color coding
    // Add fix order recommendations
}

3. Output format:

✗ Compilation failed with 5 errors

Error Summary:
  Critical: 1 error
  High: 3 errors
  Medium: 1 error

By Category:
  ❌ Schema Validation: 2 errors
  ⚠️  Permissions: 2 errors
  ⚠️  Network: 1 error

Recommended Fix Order:
  1. Fix schema errors first (typos, invalid fields)
  2. Address permission issues
  3. Configure network access

Use --verbose to see detailed error messages
See (redacted)

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-14 | Workflow Validation and Error Feedback Quality | Custom | Y | First quality improvement run - established baseline for validation and error message quality |

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Implement "Did You Mean" suggestions** - High impact, prevents user frustration from typos (Task 1) - Priority: High
2. **Add actionable hints to strict mode errors** - Quick win, improves security guidance (Task 2) - Priority: High

### Short-term Actions (This Month)
1. **Standardize error message format** - Follow time_delta.go pattern across codebase (Task 3) - Priority: Medium
2. **Create centralized error reference** - Searchable documentation for common errors (Task 4) - Priority: Medium

### Long-term Actions (This Quarter)
1. **Implement validation error summary** - Better multi-error reporting (Task 5) - Priority: Low
2. **Add interactive error resolution** - CLI prompts to fix common errors automatically
3. **Error analytics** - Track most common errors to guide UX improvements

---

## 📈 Success Metrics

Track these metrics to measure improvement in the **Workflow Validation and Error Feedback Quality** focus area:

- **Typo detection rate**: 0% → 90%+ (via fuzzy matching)
- **Error messages with examples**: ~10% → 60%+ (increase from ~24/257)
- **Error messages with actionable hints**: 0% → 40%+ (especially strict mode, permissions, schema)
- **User issue reports about confusing errors**: Baseline TBD → Reduce by 50%
- **Documentation coverage of common errors**: 0 → 15+ documented error scenarios
- **Average time to resolve compilation error**: Measure via user feedback → Reduce by 30%

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign Tasks 1 and 2 to Copilot agent via planner agent (high priority)
3. Track progress on improvement items in GitHub Projects
4. Re-evaluate this focus area in Q2 2025 to measure impact
5. Next analysis: 2025-11-15 - Focus area will be selected based on diversity algorithm (targeting 60% custom areas)

---

*Generated by Repository Quality Improvement Agent*  
*Strategy: Custom Focus Area (60% target) - Repository-specific validation and error feedback analysis*  
*Next analysis: 2025-11-15 - Will select new focus area using diversity algorithm*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19365623994)

[pelikhan]

/plan

[github-actions[bot]]

✅ Agentic Plan Command completed successfully.

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 1 week ago.