[Schema Consistency] Cross-Schema Consistency: 8 Type & Feature Mismatches Found

[github-actions[bot]]

This analysis focused on cross-schema consistency by comparing the three JSON schemas that define the gh-aw workflow configuration system. Using Strategy-004 (Cross-Schema Consistency & Type Analysis), I discovered 8 inconsistencies affecting type definitions, feature completeness, and documentation quality across the schema boundaries.

Key Statistics:

• Inconsistencies Found: 8 (3 critical, 5 moderate)
• Schemas Analyzed: 3 (main_workflow, included_file, mcp_config)
• Shared Fields: 12 with varying degrees of consistency
• Strategy Effectiveness: Very High (proven strategy, 6th successful use)

The most significant finding is that the included_file_schema.json systematically provides less feature coverage than the main schema for shared fields like engine, network, and safe-outputs, potentially limiting the capabilities of included workflow files.

Full Consistency Report

Critical Issues

1. Engine Version Field Type Mismatch 🔴

Impact: Included files cannot use numeric version values

Details:

• Main schema: Accepts type: ["string", "number"] with examples [20, 3.11, "latest"]
• Included schema: Only accepts type: "string"
• Breaking change: Users expecting engine.version: 20 (for Python) will fail validation in included files
• Example that works in main but fails in included:

  engine:
    id: claude
    version: 20  # ❌ Type error in included files

Location:

• pkg/parser/schemas/main_workflow_schema.json line 82 (approx)
• pkg/parser/schemas/included_file_schema.json line 41 (approx)

Recommendation: Change included schema to accept ["string", "number"] for consistency

---

2. Engine Configuration Missing Advanced Features 🔴

Impact: Included files cannot configure 5 critical engine properties

Missing fields in included schema:

• concurrency - GitHub Actions concurrency control (group, cancel-in-progress)
• user-agent - Custom user agent for codex engine
• error_patterns - Custom error pattern validation with regex
• config - Additional TOML configuration for codex engine
• args - CLI arguments array for engine customization

Real-world impact: Workflows that import shared configurations cannot:

• Control job concurrency at the engine level
• Customize codex engine behavior with TOML config
• Pass custom CLI arguments to engines
• Define custom error patterns for log validation

Files affected: All .github/workflows/shared/**/*.md files using engine configuration

Recommendation: Add all 5 fields to included schema or document why they're intentionally excluded

---

3. Network Firewall Configuration Missing 🔴

Impact: Cannot configure network firewall in included files

Details:

• Main schema: Has network.firewall property with version and log-level sub-fields
• Included schema: Missing firewall property entirely
• Example configuration available only in main:

  network:
    allowed: ["defaults", "github"]
    firewall:
      version: "v1.0.0"
      log-level: "debug"

Why this matters: Firewall configuration controls network access logging and version pinning for security auditing

Recommendation: Add network.firewall to included schema for feature parity

---

Moderate Issues

4. Safe-Outputs Field Structure Divergence 🟡

Finding: Main and included schemas have completely different safe-outputs structures

Main schema safe-outputs:

• 24+ operation types: create-issue, create-discussion, add-comment, create-pull-request, etc.
• Each operation fully configured with labels, assignees, max counts, etc.
• Used for AI agent output processing

Included schema safe-outputs:

• ONLY has jobs property
• References $defs/safe_job for custom job definitions
• Used for defining reusable safe-job templates

Assessment: This appears to be intentional design - different purposes

• Main: AI output operations
• Included: Custom job templates

Recommendation: Document this difference in schema comments and user documentation

---

5. Permissions Field Description Mismatch 🟡

Finding: Description suggests merging behavior not explained elsewhere

Main schema description:

"GitHub token permissions for the workflow. Controls what the GITHUB_TOKEN can access during execution..."

Included schema description:

"GitHub Actions permissions for the workflow (merged with main workflow permissions)"

Questions raised:

• How exactly are permissions merged?
• Which takes precedence - main or included?
• Are permissions additive or do they override?
• What happens with conflicting permission levels?

Recommendation: Add schema $comment explaining merge behavior or reference documentation

---

6. MCP Config Schema Validation Confusion 🟡

Finding: mcp_config_schema.json mixes stdio and HTTP requirements in anyOf

Current anyOf:

"anyOf": [
  { "required": ["type"] },
  { "required": ["url"] },      // ← HTTP-only field
  { "required": ["command"] },   // ← stdio-only field
  { "required": ["container"] }  // ← stdio-only field
]

Issue: Schema is used standalone for MCP configuration but combines validation for two distinct connection types (stdio and HTTP)

Impact: Confusing validation messages - suggests url could work with stdio

Recommendation: Split into separate schemas or use conditional validation (if/then) to separate stdio and HTTP validation paths

---

7. $defs Duplication Risk 🟡

Finding: Two schemas duplicate identical $defs without coordination

Duplicated definitions:

• stdio_mcp_tool - Appears in main and included (currently 100% identical)
• http_mcp_tool - Appears in main and included (currently 100% identical)

Current state: ✅ Definitions are identical
Risk: Future schema evolution could cause drift if one is updated without the other

Scenarios that could cause problems:

• Adding a new property to main but forgetting included
• Changing validation rules in one schema only
• Updating descriptions in one place only

Recommendation:

• Extract shared definitions to a common_defs.json file and $ref it from both schemas
• OR: Add automated tests to verify $defs consistency between schemas
• OR: Document in CONTRIBUTING.md that $defs must be kept in sync

---

8. Network Description Quality Gap 🟡

Finding: Main schema provides much richer documentation

Main schema network description:

"Network access control for AI engines using ecosystem identifiers and domain allowlists. Controls web fetch and search capabilities."

Examples: ["defaults", "python", "node", "*.example.com"]

Included schema network description:

"Network permissions configuration for allowed domains"

Impact: Developers using included files may not discover:

• Ecosystem identifiers (defaults, python, node, github)
• Wildcard domain patterns
• Integration with web fetch and search capabilities

Recommendation: Synchronize descriptions or add cross-references

---

Positive Findings ✅

The analysis revealed excellent practices in several areas:

1. MCP Tool Definitions Perfect: stdio_mcp_tool and http_mcp_tool are byte-for-byte identical across main and included schemas
2. No Broken References: All $ref pointers resolve correctly - zero broken cross-references
3. Enum Consistency: Shared enum values (e.g., engine.id) are consistent where fields overlap
4. Naming Conventions: Excellent kebab-case discipline across all schemas
5. additionalProperties: Well-maintained with 96+ constraints preventing typos

---

Schema Coverage Analysis

Field Distribution

Shared fields (12 total):

• description, engine, mcp-servers, network, permissions, runtimes
• safe-inputs, safe-outputs, secret-masking, services, steps, tools

Main schema exclusive (26 fields):

• cache, command, concurrency, container, env, environment, features
• github-token, githubActionsStep, if, imports, jobs, name, on
• post-steps, roles, run-name, runs-on, sandbox, source, strict
• timeout-minutes, timeout_minutes, tracker-id

Included schema exclusive (2 fields):

• applyTo - Glob patterns for custom agent instruction files
• inputs - Workflow input parameters for shared components

$defs Distribution

Main schema $defs (4):

1. engine_config - Referenced by engine property
2. github_token - Referenced by github-token and safe-outputs
3. http_mcp_tool - MCP HTTP connection definition
4. stdio_mcp_tool - MCP stdio connection definition

Included schema $defs (3):

1. http_mcp_tool - MCP HTTP connection definition (DUPLICATE)
2. stdio_mcp_tool - MCP stdio connection definition (DUPLICATE)
3. safe_job - Custom safe-job template (UNIQUE)

MCP config schema: No $defs (flat structure)

---

Recommendations by Priority

High Priority (Breaking Changes)

1. Fix engine.version type - Change included schema to ["string", "number"]

   • Files: pkg/parser/schemas/included_file_schema.json
   • Lines: ~41
   • Effort: 5 minutes
   • Risk: Low - strictly additive change

2. Add missing engine properties - Add concurrency, args, config, error_patterns, user-agent

   • Files: pkg/parser/schemas/included_file_schema.json
   • Lines: ~35-45
   • Effort: 30 minutes
   • Risk: Low - additive changes with optional properties

3. Add network.firewall to included schema

   • Files: pkg/parser/schemas/included_file_schema.json
   • Lines: ~70-85
   • Effort: 15 minutes
   • Risk: Low - optional property addition

Medium Priority (Documentation)

4. Document safe-outputs behavior difference

   • Add $comment fields explaining main vs included purposes
   • Update docs/reference/frontmatter.md with clarification
   • Effort: 20 minutes

5. Document permissions merging

   • Explain merge algorithm in schema $comment
   • Add examples to documentation
   • Effort: 30 minutes

6. Fix mcp_config_schema anyOf

   • Separate stdio and HTTP validation with conditional logic
   • Files: pkg/parser/schemas/mcp_config_schema.json
   • Effort: 45 minutes

Low Priority (Maintenance)

7. Create shared $defs file

   • Extract stdio_mcp_tool and http_mcp_tool to common_defs.json
   • Update both schemas to $ref the common file
   • Effort: 1-2 hours
   • Benefit: Eliminates duplication risk

8. Standardize descriptions

   • Sync network, permissions, tools descriptions between schemas
   • Effort: 30 minutes

9. Add schema version metadata

   • Add $id and version fields to all schemas
   • Effort: 10 minutes

---

Impact Assessment

Users Affected:

• All workflows using included files from .github/workflows/shared/
• Approximately 40+ workflows import shared components
• Any workflow using numeric version values in engine config

Breaking Change Risk:

• HIGH for issue rejig docs #1 (version type mismatch)
• MEDIUM for issues Add workflow: githubnext/agentics/weekly-research #2 and Add workflow: githubnext/agentics/weekly-research #3 (missing features users may expect)

Documentation Updates Required:

• docs/reference/frontmatter.md - Add included schema limitations
• docs/reference/schemas.md - Document schema purposes and differences
• Schema $comment fields - Inline documentation

Testing Required:

• Validate existing workflows with updated schemas
• Test numeric version values in included files
• Verify permissions merging behavior

---

Strategy Performance

Strategy Used: Strategy-004 (Cross-Schema Consistency & Type Analysis)
Usage Count: 6th successful execution
Previous Use: 2025-12-01 (5 days ago)

Findings Comparison:

• 2025-12-01: 7 findings (3 critical)
• 2025-12-06: 8 findings (3 critical)
• New findings: 2 (version type mismatch, missing engine properties)

Effectiveness: VERY HIGH - Continues to discover new inconsistencies
Pattern: Different findings each run due to evolving focus areas
Recommendation: Continue using every 4-5 analyses as intended

---

Next Steps

1. Immediate: Fix critical type mismatches (issues rejig docs #1-3)
2. Short-term: Document differences and merging behavior (issues Add workflow: githubnext/agentics/weekly-research #4-5)
3. Long-term: Implement shared $defs architecture to prevent drift (issue Weekly Research Report: AI Workflow Automation Landscape and Market Opportunities - August 2025 #7)

Cross-reference:

• Previous Strategy-004 run: 2025-12-01 (found different set of issues)
• Related Strategy-011: Network Configuration Analysis (2025-11-11)
• Related Strategy-008: Schema Completeness (2025-11-18)

---

Strategy Execution Notes

Strategy Selected: Cross-Schema Consistency & Type Analysis (Strategy-004)
Selection Method: Day-of-year modulo (340 mod 10 = 0, use proven strategy)
Execution Time: ~8 minutes
Tools Used: jq, bash diff, schema comparison scripts
Cache Updated: ✅ strategies.json updated with new findings

Methodology:

1. Extracted all property keys from three schemas
2. Compared $defs definitions byte-by-byte
3. Analyzed shared field type definitions
4. Cross-referenced descriptions and examples
5. Validated consistency of validation rules (anyOf, oneOf, required)

Files Analyzed:

• pkg/parser/schemas/main_workflow_schema.json (2,847 lines)
• pkg/parser/schemas/included_file_schema.json (1,243 lines)
• pkg/parser/schemas/mcp_config_schema.json (154 lines)

Total Schema Size: 4,244 lines of JSON Schema definitions

AI generated by Schema Consistency Checker

[pelikhan]

/plan

[github-actions[bot]]

🎉 Yo ho ho! Plan Command found the treasure and completed successfully! ⚓💰

[pelikhan]

/plan

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Schema Consistency Checker.
🗺️ A newer treasure map awaits ye at Discussion #5708.
Fair winds, matey! 🏴‍☠️