[Schema Consistency] 🔍 Schema Consistency Check - Edge Case Compilation Testing (2025-11-25)

[github-actions[bot]]

Executive Summary

This analysis used a novel edge case compilation testing strategy to identify validation gaps in the workflow schema. By creating 25 test workflows with boundary values and malformed inputs, then compiling them with gh-aw compile, I discovered 10 critical validation gaps where the schema accepts illogical or problematic values that should be rejected.

Key Finding: The schema is missing essential numeric range constraints (minimum/maximum), array size constraints (minItems), and pattern validation for structured strings like cron expressions.

Good News: No production workflows use any of these problematic edge cases, indicating healthy code practices. The parser's type validation is also stricter than previously believed - it correctly rejects string values for integer fields at parse time.

Critical Validation Gaps (10 Issues Found)

1. Negative Timeout Values ❌

Test: timeout-minutes: -10
Expected: Should fail (negative timeout is illogical)
Actual: ✓ PASSED (compiled successfully)
Impact: Workflows can specify negative timeouts which have no semantic meaning
Fix: Add minimum: 1 constraint to timeout-minutes in schema

Schema Location: pkg/parser/schemas/main_workflow_schema.json:timeout-minutes

---

2. Zero Timeout ❌

Test: timeout-minutes: 0
Expected: Should fail (0 timeout = instant termination)
Actual: ✓ PASSED (compiled successfully)
Impact: Workflows with zero timeout would terminate immediately
Fix: Add minimum: 1 constraint to timeout-minutes in schema

---

3. Extremely Large Timeout ⚠️

Test: timeout-minutes: 999999999
Expected: Should fail or warn (exceeds GitHub Actions practical limits)
Actual: ✓ PASSED (compiled successfully)
Impact: Unrealistic timeout values accepted without validation
Fix: Add maximum: 2160 (36 hours, GitHub Actions max) to timeout-minutes

Reference: GitHub Actions has a maximum timeout of 2160 minutes (36 hours)

---

4. Invalid Cron Syntax ❌

Test: schedule[].cron: "invalid cron syntax"
Expected: Should fail (invalid cron format)
Actual: ✓ PASSED (compiled successfully)
Impact: Workflows with invalid cron expressions fail at runtime, not compile time
Fix: Add pattern constraint for cron format validation

Suggestion: Pattern should match standard cron syntax: ^(\S+\s+){4}\S+$ or more specific regex

---

5. Empty Schedule Array ❌

Test: on: { schedule: [] }
Expected: Should fail (empty schedule is meaningless)
Actual: ✓ PASSED (compiled successfully)
Impact: Workflows with empty schedules will never run
Fix: Add minItems: 1 constraint to schedule array

---

6. Empty runs-on Array ❌

Test: runs-on: []
Expected: Should fail (no runner specified)
Actual: ✓ PASSED (compiled successfully)
Impact: Workflow would fail at runtime with no runner specified
Fix: Add minItems: 1 constraint to runs-on array (when array form is used)

---

7. Empty Concurrency Group ❌

Test: concurrency: { group: "" }
Expected: Should fail (empty group name defeats purpose)
Actual: ✓ PASSED (compiled successfully)
Impact: Empty concurrency group provides no actual concurrency control
Fix: Add minLength: 1 constraint to concurrency.group

---

8. Both branches and branches-ignore ⚠️

Test: push: { branches: ["main"], branches-ignore: ["dev"] }
Expected: Should fail or warn (mutually exclusive in GitHub Actions)
Actual: ✓ PASSED (compiled successfully)
Impact: GitHub Actions behavior is undefined when both filters are present
Fix: Add schema constraint or validation rule for mutual exclusivity, or document expected behavior

Note: This may require conditional validation beyond JSON Schema capabilities

---

9. runs-on Group Without Labels ⚠️

Test: runs-on: { group: "my-group" } (no labels field)
Expected: Unclear (depends on whether labels is required)
Actual: ✓ PASSED (compiled successfully)
Impact: May work if group alone is sufficient, but lacks documentation
Fix: Document whether labels is optional when using group, or add schema requirement

---

10. Logically Contradictory Filters ⚠️

Test: pull_request: { draft: true, types: ["ready_for_review"] }
Expected: Should warn (logically contradictory)
Actual: ✓ PASSED (compiled successfully)
Impact: Draft PRs cannot be ready for review - filter will never match
Fix: Add conditional validation warning or documentation about incompatible filter combinations

Validations Working Correctly (12 Confirmations)

Parser Validation Strengths ✓

The following tests confirm that the parser correctly enforces schema constraints:

1. Empty Name (Test 02): ✗ Correctly rejects name: "" (minLength: 1)
2. Short tracker-id (Test 03): ✗ Correctly enforces minLength: 8 for tracker-id
3. Invalid tracker-id Pattern (Test 04): ✗ Correctly validates pattern ^[a-zA-Z0-9_-]+$
4. String Timeout (Test 07): ✗ Correctly rejects timeout-minutes: "300" (expects integer)
5. Float Timeout (Test 08): ✗ Correctly rejects timeout-minutes: 45.7 (expects integer)
6. Unknown Field (Test 09): ✗ Correctly rejects unknown fields (additionalProperties: false works)
7. Invalid Permission (Test 10): ✗ Correctly validates permission enum values
8. Invalid Reaction (Test 11): ✗ Correctly validates reaction enum values
9. Duplicate YAML Keys (Test 12): ✗ Parser detects duplicate on: keys
10. Empty on Field (Test 13): ✗ Correctly rejects empty string for on field
11. Empty Command (Test 15): ✗ Correctly validates command minLength: 1
12. Invalid PR Type (Test 25): ✗ Correctly validates PR types enum

Important Finding: Type Validation is Stricter Than Believed

Previous Claim (Strategy-017): "String-to-number type coercion not documented - timeout accepts '300' strings"

Test 07 Result: ✗ Parser correctly rejects string "300" for timeout-minutes

Conclusion: The parser does NOT accept strings for integer fields at parse time. Type coercion may happen elsewhere in the system, but frontmatter parsing enforces strict type matching per JSON Schema.

Schema Improvement Recommendations

High Priority (Security/Reliability Impact)

1. Add Numeric Range Constraints

File: pkg/parser/schemas/main_workflow_schema.json

{
  "timeout-minutes": {
    "type": "integer",
    "minimum": 1,
    "maximum": 2160,
    "description": "Timeout in minutes (1-2160, max 36 hours per GitHub Actions limit)"
  }
}

Apply to: All integer fields representing durations, counts, limits, etc.

2. Add Array Size Constraints

{
  "schedule": {
    "type": "array",
    "minItems": 1,
    "items": { ... }
  },
  "runs-on": {
    "oneOf": [
      { "type": "string" },
      { 
        "type": "array",
        "minItems": 1,
        "items": { "type": "string" }
      },
      { "type": "object", ... }
    ]
  }
}

Apply to: schedule, runs-on (array form), and other arrays where empty is meaningless

3. Add String Length Constraints

{
  "concurrency": {
    "oneOf": [
      { "type": "string", "minLength": 1 },
      {
        "type": "object",
        "properties": {
          "group": { "type": "string", "minLength": 1 }
        }
      }
    ]
  }
}

Apply to: concurrency.group and other required identifier strings

4. Add Pattern Validation for Structured Strings

{
  "cron": {
    "type": "string",
    "pattern": "^(\\S+\\s+){4}\\S+$",
    "description": "Cron expression with 5 fields (minute hour day month weekday)"
  }
}

Note: Consider using a more comprehensive cron validation pattern or runtime validation

Medium Priority (Documentation/UX Impact)

5. Document Mutual Exclusivity

Add to schema descriptions or $comment fields:

• branches and branches-ignore are mutually exclusive
• paths and paths-ignore are mutually exclusive
• tags and tags-ignore are mutually exclusive

Consider conditional validation if JSON Schema supports it, or add runtime validation.

6. Document Logical Conflicts

Add warnings in schema descriptions:

• draft: true is incompatible with types: ["ready_for_review"]
• Other logically contradictory filter combinations

7. Clarify Optional Fields

Document in schema description for runs-on.group:

• Whether labels field is required when group is specified
• What happens if only group is provided without labels

Real-World Impact Assessment

Production Workflow Analysis

✅ GOOD NEWS: No production workflows use any of the problematic edge cases identified:

• ✅ No timeout-minutes: 0 found
• ✅ No timeout-minutes: -N (negative) found
• ✅ No schedule: [] (empty) found
• ✅ No runs-on: [] (empty) found
• ✅ All timeout values in production are reasonable (10-30 minutes typically)

Sample from .github/workflows/*.md:

• archie.md: timeout-minutes: 10
• artifacts-summary.md: timeout-minutes: 15
• audit-workflows.md: timeout-minutes: 30
• blog-auditor.md: timeout-minutes: 10
• brave.md: timeout-minutes: 10

Risk Level: Low to Medium

Current Risk: Low - No production workflows affected by validation gaps

Future Risk: Medium - New workflows could accidentally use invalid values and fail at runtime instead of compile time

Recommendation: Implement schema improvements to catch issues at design time (IDE validation) rather than runtime

Methodology & Strategy Details

Strategy: Edge Case Compilation Testing

ID: strategy-020
Name: Edge Case Compilation Testing & Boundary Value Analysis
Type: Novel (30% exploration - day 329 mod 10 = 9)

Approach

1. Schema Analysis: Extract validation constraints from main_workflow_schema.json
2. Test Creation: Create 25 test workflows targeting edge cases:
   • Boundary values (0, negative, extremely large)
   • Empty collections (arrays, strings)
   • Invalid formats (cron syntax, enum values)
   • Type mismatches (strings for integers, floats for integers)
   • Unknown fields
   • Logical contradictions
3. Compilation Testing: Compile each workflow with gh-aw compile command
4. Result Analysis: Categorize as "should reject but passed" vs "correctly rejected"
5. Real-World Check: Verify if any production workflows use problematic patterns

Tools Used

• jq: Schema parsing and field extraction
• gh-aw compile: Actual compilation testing (not simulation)
• grep: Production workflow analysis
• Manual analysis: Logical contradiction detection

Test Coverage Matrix

Category	Tests	Pass	Fail	Coverage
Numeric bounds	5	3 ❌	2 ✅	100%
Type validation	3	0 ✅	3 ✅	100%
Array constraints	2	2 ❌	0	100%
String validation	4	1 ❌	3 ✅	100%
Enum validation	3	0 ✅	3 ✅	100%
Logical conflicts	2	2 ⚠️	0	100%
Format validation	1	1 ❌	0	100%
Unknown fields	1	0 ✅	1 ✅	100%
YAML parsing	1	0 ✅	1 ✅	100%
Null handling	2	1 ✅	1 ✅	100%

Legend:

• ✅ = Correctly rejected (expected behavior)
• ❌ = Incorrectly accepted (validation gap)
• ⚠️ = Accepted but questionable (needs documentation)

Key Insights

1. Parse-Time vs Runtime Validation: Schema validation happens at parse time, security validation (strict mode) happens at compile time
2. Type Strictness: Parser is stricter than previously believed - correctly rejects type mismatches
3. Missing Constraints: Schema lacks many numeric range, array size, and format constraints
4. Production Quality: No production workflows use invalid patterns (good practices)

Comparison with Previous Strategies

Strategy-017 Claim Verification

Strategy-017 Claim: "String-to-number type coercion not documented in schema (timeout accepts '300' strings)"

Test 07 (This Analysis): timeout-minutes: "300"

• Result: ✗ FAILED compilation
• Error: got string, want integer
• Conclusion: Parser correctly rejects strings for integer fields

Updated Understanding:

• Frontmatter parsing enforces strict types per JSON Schema
• Type coercion (if it exists) happens later in compilation or runtime, not at parse time
• Strategy-017 may have tested runtime behavior, not parse-time validation

Unique Contributions of This Strategy

Novel Aspects:

1. Actual Compilation Testing: Uses real gh-aw compile instead of static analysis
2. Boundary Value Focus: Tests edge cases (0, negative, huge numbers, empty arrays)
3. Logical Contradiction Testing: Tests semantically invalid combinations
4. Validation Gap Discovery: Finds missing schema constraints (min/max, minItems, patterns)
5. Cross-Strategy Verification: Confirms or refutes previous strategy findings

Complements:

• Strategy-003: Required field enforcement (this adds boundary validation)
• Strategy-012: Dynamic validation (this adds edge case testing)
• Strategy-017: Type coercion (this verifies parse-time type strictness)

Next Steps & Action Items

Immediate Actions

1. Schema Updates (High Priority)

File: pkg/parser/schemas/main_workflow_schema.json

• Add minimum: 1, maximum: 2160 to timeout-minutes
• Add minimum: 1, maximum: 2160 to timeout_minutes (legacy field)
• Add minItems: 1 to schedule array
• Add minItems: 1 to runs-on array (in oneOf array variant)
• Add minLength: 1 to concurrency.group
• Add cron pattern validation to schedule[].cron

2. Documentation Updates

Files: docs/src/content/docs/reference/frontmatter*.md

• Document GitHub Actions timeout maximum (2160 minutes / 36 hours)
• Document mutual exclusivity of branches/branches-ignore filters
• Document logical conflicts (draft + ready_for_review)
• Clarify runs-on.group - whether labels is optional
• Add examples showing invalid configurations and error messages

3. Validation Enhancements (Medium Priority)

Consider:

• Runtime warnings for extremely large timeout values (even if within max)
• Compile-time warnings for logically contradictory filter combinations
• Better cron syntax validation (potentially at compile time if schema patterns insufficient)

4. Testing Infrastructure

• Add these 25 test workflows to automated test suite (negative tests)
• Create CI check for schema constraint completeness
• Periodically re-run edge case compilation tests to catch regressions

Long-Term Improvements

Schema Validation Framework

Consider implementing:

1. Conditional Validation: Rules like "if branches exists, branches-ignore cannot"
2. Semantic Validation: Logical contradiction checking beyond JSON Schema
3. Range Recommendations: Suggest typical values, warn on outliers
4. Context-Aware Validation: Engine-specific constraints (building on strategy-005 findings)

Parser Enhancements

Potential improvements:

1. Warning System: Non-fatal warnings for questionable but technically valid configs
2. Validation Levels: --strict flag to enable additional checks
3. Validation Report: Detailed explanation of what was validated and what was skipped

Strategy Performance

Strategy Used: Edge Case Compilation Testing & Boundary Value Analysis (strategy-020)
Findings: 10 critical validation gaps
Effectiveness: Very High
Should Reuse: Yes
Recommendation: Use every 6-8 analyses to test schema constraint completeness

Unique Value: Only strategy to actually compile workflows with edge cases. Reveals missing schema constraints invisible to static analysis. Verifies claims from previous strategies through empirical testing.

Best Paired With:

• Strategy-003: Required field enforcement
• Strategy-012: Dynamic validation infrastructure
• Strategy-017: Type coercion analysis (this provides corrections)

Conclusion

The schema validation is strong for documented constraints (type checking, enums, patterns) but missing essential boundary constraints (min/max, minItems, minLength). The good news is that production workflows don't use any problematic edge cases, indicating healthy development practices. However, adding the recommended schema constraints would improve the IDE experience and catch configuration errors at design time rather than runtime.

Priority: Medium - No urgent production issues, but important for preventing future problems and improving developer experience.

AI generated by Schema Consistency Checker

[github-actions[bot]]

⚓ Avast! This discussion be marked as outdated by Schema Consistency Checker.
🗺️ A newer treasure map awaits ye at Discussion #5254.
Fair winds, matey! 🏴‍☠️