🎯 Repository Quality Improvement Report - Workflow Input/Output Contract Validation

[github-actions[bot]]

🎯 Repository Quality Improvement Report - Workflow Input/Output Contract Validation

Analysis Date: 2025-12-05
Focus Area: Workflow Input/Output Contract Validation
Strategy Type: Custom
Custom Area: Yes - This focus area addresses gh-aw's unique challenge of ensuring type safety and contract adherence in the workflow compilation pipeline, where markdown frontmatter is transformed into GitHub Actions workflows with safe output configurations.

Executive Summary

The gh-aw project compiles 152 markdown workflows into GitHub Actions with 98 safe output types, but lacks a formal input/output contract system. While the codebase has strong JSON schema validation (5,850 LOC across 3 schemas with 684 documented properties) and 69 typed config structs, there's no systematic validation of workflow inputs/outputs at compilation time or runtime. This gap creates potential for type mismatches, missing parameter validation, and unclear contracts between workflow authors and safe output processors.

Key findings: 26 of 98 non-test JavaScript safe output files (27%) lack JSDoc type annotations, making contract verification difficult. The schema infrastructure is robust but doesn't enforce input/output contracts for workflows. Only 17 explicit type validation patterns exist despite 69 config structs requiring validation. This custom analysis reveals an opportunity to introduce contract-first workflow development with compile-time validation.

Full Analysis Report

Focus Area: Workflow Input/Output Contract Validation

Rationale for This Custom Focus Area

Unlike standard quality categories (testing, security, documentation), this focus area is uniquely specific to gh-aw's architecture:

1. Workflow Compilation Pipeline: gh-aw transforms markdown frontmatter into GitHub Actions YAML with embedded JavaScript safe outputs
2. Type Safety Gap: The boundary between Go compilation code and JavaScript runtime lacks formal contract validation
3. Safe Output Ecosystem: 98 safe output types process workflow data, but contracts between them are implicit
4. Schema vs. Runtime Mismatch: Strong schema validation at parse time doesn't guarantee runtime type correctness
5. Developer Experience: Workflow authors need clear contracts to understand what inputs/outputs are available and their types

This analysis examines the entire contract validation lifecycle: frontmatter parsing → compilation → safe output processing → runtime execution.

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total markdown workflows	152	✅
Compiled .lock.yml files	106	⚠️ 46 uncompiled
Safe output CJS files	98 (non-test)	✅
CJS files with JSDoc	72 (73%)	⚠️ 27% missing
Go config struct types	69	✅
JSON schema files	3 (5,850 LOC)	✅
Schema documented properties	684	✅ Excellent
Type validation patterns	17	❌ Low coverage
Validation test files	39	✅
Validation error messages	22	⚠️
Config validation patterns	47	✅
Reflection-based checks	0	✅ No runtime overhead

Findings

Strengths

• Comprehensive JSON schemas: 5,850 lines covering workflow frontmatter with 684 documented properties
• Strong Go type system: 69 typed config structs provide compile-time safety on the Go side
• Good JSDoc adoption: 73% of safe output files have type annotations
• Validation infrastructure: 39 test files dedicated to validation, 47 config validation patterns
• Zero reflection overhead: No runtime type reflection, keeping performance high
• Robust schema tests: Comprehensive schema validation test coverage

Areas for Improvement

1. Missing workflow input/output system (Critical)

   • Zero workflows define inputs: or outputs: in frontmatter
   • No schema support for workflow-level inputs/outputs
   • No compile-time validation of parameter passing between workflows
   • Severity: High - Limits workflow reusability and composability

2. Incomplete JSDoc coverage (High)

   • 26 safe output files (27%) lack @param and @returns annotations
   • Makes contract verification difficult for JavaScript processors
   • No automated enforcement of JSDoc requirements
   • Severity: High - Creates maintenance burden and integration risks

3. Implicit type contracts (Medium)

   • Only 17 explicit type validation patterns despite 69 config structs
   • Type safety relies on Go struct tags rather than explicit validation
   • No contract testing between Go compilation and JS execution
   • Severity: Medium - May cause runtime type errors

4. 46 uncompiled workflows (Medium)

   • 30% of markdown workflows lack corresponding .lock.yml files
   • Unclear if these are intentionally uncompiled or stale
   • May indicate workflow authoring friction
   • Severity: Medium - Workflow ecosystem health concern

5. Safe output contract documentation (Low)

   • Limited documentation of safe output input/output contracts
   • No central registry of safe output types and their schemas
   • Workflow authors must read CJS source to understand contracts
   • Severity: Low - Developer experience issue

Detailed Analysis

1. Workflow Input/Output System Gap

The JSON schema infrastructure is excellent for validating frontmatter structure, but there's no support for defining workflow inputs and outputs that can be passed between composed workflows. The imports field allows workflow composition, but there's no type-safe way to pass parameters.

Current State:

# No inputs/outputs defined
imports:
  - shared/jqschema.md
  - shared/reporting.md

Desired State:

inputs:
  repository:
    type: string
    required: true
    description: Target repository
  issue_count:
    type: number
    default: 10
    
outputs:
  created_discussion_url:
    type: string
    description: URL of created discussion

2. Safe Input Parameter Validation

The SafeInputParam struct exists (7 references in safe_inputs.go) with type definitions, but there's no compile-time validation that JavaScript implementations match declared types:

type SafeInputParam struct {
    Type        string // JSON schema type (string, number, boolean, array, object)
    Description string
    Required    bool
    Default     any
}

The gap: No validation that the JavaScript script:, run:, or py: implementations correctly handle the declared parameter types.

3. JavaScript Safe Output Type Contracts

While 73% of safe outputs have JSDoc, the annotations aren't enforced or validated during compilation:

Good Example (create_discussion.cjs):

/**
 * Fetch repository ID and discussion categories for a repository
 * `@param` {string} owner - Repository owner
 * `@param` {string} repo - Repository name
 * `@returns` {Promise<{repositoryId: string, discussionCategories: Array<...>}|null>}
 */

Missing Annotations:
26 files lack these annotations, making contract verification manual and error-prone.

4. Uncompiled Workflow Analysis

46 workflows (30%) exist as .md files without corresponding .lock.yml files. This could indicate:

• Incomplete workflow development
• Intentionally uncompiled shared includes
• Stale workflow files
• Compilation errors not caught by CI

Need systematic audit to determine intent and ensure consistency.

5. Type Validation Pattern Analysis

Only 17 explicit type validation patterns exist for 69 config structs (25% coverage). Most validation is implicit via Go's type system and struct tags. While this provides basic type safety, it doesn't catch:

• Invalid enum values
• Range constraints
• Cross-field validations
• Complex nested structure requirements

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

The following code regions and tasks should be processed by the Copilot agent. Each section is marked for easy identification by the planner agent.

Task 1: Add Workflow Input/Output Schema Support

Priority: High
Estimated Effort: Large
Focus Area: Workflow Contract Validation

Description:

Extend the JSON schema in pkg/parser/schemas/main_workflow_schema.json to support workflow-level inputs and outputs definitions. This will enable type-safe parameter passing between composed workflows and provide compile-time validation of workflow contracts.

Acceptance Criteria:

• Add inputs property to main workflow schema with per-input type, description, required, and default fields
• Add outputs property to main workflow schema with per-output type and description fields
• Implement Go parsing logic in pkg/parser/frontmatter.go to extract and validate input/output definitions
• Add validation that ensures imported workflows with inputs: configuration receive required parameters
• Create unit tests for input/output schema validation covering all JSON schema types
• Update at least 3 example workflows to demonstrate input/output usage
• Document input/output system in schema descriptions

Code Region: pkg/parser/schemas/main_workflow_schema.json, pkg/parser/frontmatter.go

Add formal input/output contract support to workflow frontmatter schema and parsing logic. Enable workflows to declare typed inputs (with required/optional and defaults) and outputs, similar to GitHub Actions reusable workflows. Implement validation in the parser to ensure:

1. Input types are valid JSON schema types (string, number, boolean, array, object)
2. Required inputs are provided when workflows are imported with `inputs:` configuration
3. Output declarations specify type and description
4. Compile-time warnings for missing required parameters

Reference the existing `SafeInputParam` struct in `pkg/workflow/safe_inputs.go` as a model for parameter definitions. Extend the import validation logic to check input contracts.

---

Task 2: Enforce JSDoc Type Annotations for Safe Outputs

Priority: High
Estimated Effort: Medium
Focus Area: Safe Output Contract Documentation

Description:

Create a validation script and CI check to ensure all safe output JavaScript files have complete JSDoc type annotations. Add missing annotations to the 26 files currently lacking them.

Acceptance Criteria:

• Create validation script in scripts/validate-jsdoc.sh that checks for @param and @returns in all .cjs files
• Add JSDoc annotations to all 26 files missing them (prioritize the 10 most commonly used safe outputs)
• Add Makefile target make lint-jsdoc that runs the validation
• Integrate validation into make agent-finish pipeline
• Create JSDoc template documentation in docs/ for safe output authors
• Ensure all core.setOutput() calls have corresponding @returns documentation

Code Region: pkg/workflow/js/*.cjs, scripts/, Makefile

Audit all JavaScript safe output files to identify missing JSDoc type annotations. For each file without complete `@param` and `@returns` annotations:

1. Analyze the function signatures and `core.getInput()` calls to determine parameter types
2. Examine `core.setOutput()` calls to determine return types
3. Add comprehensive JSDoc blocks with correct TypeScript-style type syntax
4. Include parameter descriptions that explain purpose and valid values

Create a bash script that:
- Finds all `.cjs` files excluding tests
- Checks for presence of JSDoc comments with `@param`/`@returns`
- Reports files missing annotations with line numbers
- Exits non-zero if any files lack complete annotations

Add this script to the Makefile and CI pipeline.

---

Task 3: Implement Safe Output Contract Registry

Priority: Medium
Estimated Effort: Medium
Focus Area: Safe Output Discovery and Documentation

Description:

Create an automated system that extracts JSDoc type information from safe output files and generates a machine-readable registry. This enables compile-time contract validation and provides a single source of truth for safe output types.

Acceptance Criteria:

• Create script in scripts/generate-safe-output-registry.go that parses JSDoc from all .cjs files
• Generate JSON registry file at pkg/workflow/safe_outputs_registry.json with schema for each safe output
• Include input parameters (name, type, required, description) and outputs in registry
• Add Makefile target make generate-registry that runs the script
• Integrate registry generation into make recompile workflow
• Create Go function to validate safe output configurations against registry at compile time
• Add test that ensures registry is up-to-date (fails if CJS files changed but registry wasn't regenerated)

Code Region: scripts/generate-safe-output-registry.go, pkg/workflow/, Makefile

Build a contract registry system for safe outputs:

1. Create a Go program that:
   - Scans `pkg/workflow/js/*.cjs` files
   - Extracts JSDoc `@param` and `@returns` annotations using regex or AST parsing
   - Generates a JSON registry mapping safe output names to their contracts
   - Includes input parameter schemas (type, description, required/optional)
   - Includes output schemas from `@returns` annotations

2. Registry JSON format:
```json
{
  "create_discussion": {
    "inputs": {
      "title": { "type": "string", "required": true, "description": "..." },
      "body": { "type": "string", "required": true, "description": "..." }
    },
    "outputs": {
      "discussion_url": { "type": "string", "description": "..." }
    }
  }
}

3. Use the registry in compilation to validate that workflow frontmatter safe-outputs configurations match expected contracts.

---

#### Task 4: Add Compile-Time Type Validation for Safe Output Configs

**Priority**: Medium  
**Estimated Effort**: Medium  
**Focus Area**: Compilation-Time Contract Enforcement

**Description:**

Enhance the workflow compiler to validate safe output configurations against their declared contracts at compile time, using the safe output registry. This catches type mismatches and missing required parameters before runtime.

**Acceptance Criteria:**
- [ ] Add validation function in `pkg/workflow/compiler.go` that checks safe output configs against registry
- [ ] Detect when required safe output parameters are missing from workflow configuration
- [ ] Detect type mismatches (e.g., string provided where number expected)
- [ ] Generate helpful error messages using `pkg/console` formatting with suggestions
- [ ] Add 10+ test cases covering various validation scenarios
- [ ] Ensure validation errors include file path, line number, and concrete fix suggestions
- [ ] Make validation errors non-fatal with `--strict` flag to enforce them

**Code Region:** `pkg/workflow/compiler.go`, `pkg/workflow/safe_output_validation.go`

```markdown
Implement compile-time contract validation for safe outputs:

1. During workflow compilation, after parsing frontmatter, validate each safe-outputs configuration:
   - Load the safe output registry
   - For each configured safe output (e.g., `create-discussion`), look up its contract
   - Check that all required parameters are provided in the workflow config
   - Validate parameter types match expected types (use reflection or type assertions)
   - Check that parameter values are valid (e.g., enums, ranges)

2. Generate actionable error messages:

Error: Safe output 'create-discussion' missing required parameter 'title'

In: .github/workflows/quality-improvement.md
Line: 15

Expected configuration:
safe-outputs:
create-discussion:
title: "Discussion Title"
body: "Discussion Body"

3. Add `--strict` flag to make validation fatal (default: warnings only for backward compatibility)

4. Comprehensive test coverage for validation logic

---

Task 5: Audit and Classify Uncompiled Workflows

Priority: Low
Estimated Effort: Small
Focus Area: Workflow Ecosystem Health

Description:

Systematically identify the 46 markdown workflows without corresponding .lock.yml files and classify them as: intentional includes (should not be compiled), incomplete (need compilation), or stale (should be removed). Document findings and add metadata to indicate compilation intent.

Acceptance Criteria:

• Create script scripts/audit-uncompiled-workflows.sh that lists all .md files without .lock.yml
• Manually review each uncompiled workflow and classify into categories
• Add frontmatter field compile: false to intentional includes to document intent
• Compile any incomplete workflows that should have .lock.yml files
• Move or archive stale workflows that are no longer needed
• Update workflow schema to support optional compile field
• Document findings in issue or discussion with recommendations
• Add CI check that warns about uncompiled workflows without compile: false flag

Code Region: .github/workflows/, pkg/parser/schemas/main_workflow_schema.json, scripts/

Audit workflow compilation status:

1. Create bash script that compares `.md` and `.lock.yml` files:
   ```bash
   #!/bin/bash
   for md in .github/workflows/*.md; do
     lock="${md%.md}.lock.yml"
     if [ ! -f "$lock" ]; then
       echo "Uncompiled: $md"
     fi
   done

2. For each uncompiled workflow, determine:

   • Is it a shared include meant to be imported (e.g., shared/*.md)?
   • Is it incomplete and should be compiled?
   • Is it stale and should be removed?

3. Add compile: false frontmatter field to shared includes:

   ---
   name: Shared JQ Schema Tools
   compile: false  # Include only, not a standalone workflow
   ---

4. Update schema to make compile field optional boolean

5. Document findings in a report with recommendations for each uncompiled workflow

---

## 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2025-11-25 | workflow-observability-debugging | Custom | Y | Analysis of logging and debugging tools |
| 2025-11-26 | workflow-compilation-consistency | Custom | Y | .lock.yml freshness automation |
| 2025-11-27 | ci-cd | Standard | N | GitHub Actions pipeline efficiency |
| 2025-11-28 | mcp-integration-quality | Custom | Y | MCP server integration patterns |
| 2025-12-01 | testing | Standard | N | Test coverage analysis (222% Go, 219% JS) |
| 2025-12-02 | safe-output-system-reliability-validation | Custom | Y | Validation pipeline and message system |
| 2025-12-03 | testing | Standard | N | Test quality follow-up (5 critical files >1K LOC) |
| 2025-12-04 | security | Standard | N | Comprehensive security audit (99.9% pinned actions) |

**Statistics:**
- Total runs: 9 (including this run)
- Custom rate: 56% (5 custom, 4 standard)
- Unique areas explored: 8
- Most impactful: testing, safe-output-system, security

</details>

---

## 🎯 Recommendations

### Immediate Actions (This Week)
1. **Task 2: Enforce JSDoc annotations** - Priority: High - Quick wins for contract documentation
2. **Task 4: Add compile-time validation** - Priority: High - Catches errors early in development cycle

### Short-term Actions (This Month)
1. **Task 1: Add input/output schema support** - Priority: High - Enables workflow composability
2. **Task 3: Create safe output registry** - Priority: Medium - Foundation for tooling and validation

### Long-term Actions (This Quarter)
1. **Task 5: Audit uncompiled workflows** - Priority: Low - Improves repository hygiene
2. **Contract-first development culture** - Establish patterns where contracts are defined before implementation

---

## 📈 Success Metrics

Track these metrics to measure improvement in **Workflow Input/Output Contract Validation**:

- **JSDoc Coverage**: 73% → 100% (all safe outputs documented)
- **Compilation Validation**: 0 → 100% (all workflows validated against contracts)
- **Unclassified Workflows**: 46 → 0 (all workflows classified with intent documented)
- **Contract Registry**: 0 → 1 (automated registry generated from JSDoc)
- **Input/Output System**: 0 → ≥5 (workflows using typed inputs/outputs)
- **Type Validation Patterns**: 17 → 69+ (one per config struct)

---

## Next Steps

1. Review and prioritize the tasks above
2. Assign Task 1 and Task 2 (high priority) to Copilot agent via planner agent
3. Track progress on improvement items in project board
4. Re-evaluate workflow contract system in 2-3 weeks after initial implementations
5. Next quality analysis: 2025-12-06 - Focus area will be selected using diversity algorithm

---

*Generated by Repository Quality Improvement Agent*  
*Next analysis: 2025-12-06 - New custom focus area to be selected (60% probability)*


> AI generated by [Repository Quality Improvement Agent](https://github.com/githubnext/gh-aw/actions/runs/19964043210)

[github-actions[bot]]

This discussion was automatically closed because it was created by an agentic workflow more than 3 days ago.