[syntax-error-quality] Compiler error messages double-format with wrong line number (file:1:1 prefix)

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-04-03
Test Cases: 3
Average Score: 67/100
Status: ⚠️ Needs Improvement

---

Executive Summary

The compiler emits a spurious outer file:1:1: error: prefix wrapping already-formatted error messages. This causes all three error categories to display duplicate file locations — the outer one pointing to line 1 (wrong) and the inner one pointing to the actual error location (correct). IDEs that parse file:line:col: format will navigate users to line 1 instead of the actual error. Additionally, YAML syntax errors lack corrective examples showing the valid form.

Key Findings:

• Strengths: Translated messages are clear (YAML), "Did you mean?" + example for engine typos, constraint translation for numeric errors
• Weaknesses: Double-format prefix on ALL errors; YAML errors have no "correct usage" example; timeout errors lack documentation links
• Critical Issue: The file:1:1: outer wrapper makes IDE error navigation unreliable for every compile error

---

Root Cause

isFormattedCompilerError() in pkg/workflow/compiler_error_formatter.go only detects *wrappedCompilerError and *parser.FormattedParserError types. However, two code paths return pre-formatted errors as plain errors.New() / fmt.Errorf() values that evade this check:

• pkg/parser/schema_compiler.go:357,385 — validateWithSchemaAndLocation returns errors.New(formattedErr)
• pkg/workflow/frontmatter_error.go:76,80 — createFrontmatterError returns fmt.Errorf("%s\n%s", ...)

Both already produce file:line:col: error: message output. But compiler.go (line 92–96) then wraps them again with formatCompilerError(markdownPath, "error", err.Error(), err), appending a second file:1:1: error: prefix.

---

Test Case Results

Test Case 1: YAML Syntax Error (`engine claude` — missing colon) — Score: 56/100 ⚠️

Test Configuration

Workflow: audit-workflows.md (93 lines — simple workflow)
Error Type: Category A — Invalid YAML syntax
Error Introduced: Line 12: engine claude (colon removed; should be engine: claude)

Actual Compiler Output

test-1.md:1:1: error: test-1.md:13:8: error: missing ':' after key — YAML mapping entries require 'key: value' format
  13 | engine claude
           ^
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 15/25 | Acceptable |
| Actionability | 16/25 | Good |
| Context | 15/20 | Good |
| Examples | 3/15 | Poor |
| Consistency | 7/15 | Inconsistent |
| **Total** | **56/100** | **Poor** |

#### Strengths
- ✅ Translated message is clear: "missing ':' after key — YAML mapping entries require 'key: value' format"
- ✅ Source line with `^` pointer is shown (from goccy YAML)
- ✅ Correct inner line:col (`13:8`)

#### Weaknesses
- ❌ Outer `test-1.md:1:1:` prefix is wrong — IDE navigates to line 1 instead of 13
- ❌ No example of correct syntax (`engine: claude`)
- ❌ No documentation link
- ❌ Confusing dual-location format (`file:1:1:` outer + `[13:8]` inner)

#### Improvement Suggestions

1. **Fix double-wrapping**: `createFrontmatterError` should return a `*parser.FormattedParserError` so `isFormattedCompilerError` recognises it
2. **Add corrective example**: After the translation, append `\n\nCorrect: engine: claude`
3. **Single location**: Eliminate the `file:1:1:` outer wrapper

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name (`engine: copiilot`)</b> — Score: 75/100 ✅</summary>

#### Test Configuration

**Workflow**: `discussion-task-miner.md` (332 lines — medium workflow)
**Error Type**: Category B — Invalid engine name (typo)
**Error Introduced**: Line 17: `engine: copiilot` (should be `engine: copilot`)

#### Actual Compiler Output

```
test-2.md:1:1: error: test-2.md:17:9: error: 'engine' (line 17, col 9): value must be one of 'claude', 'codex', 'copilot', 'gemini'. Did you mean: copilot? Valid engines are: copilot, claude, codex. Example: engine: copilot
  17 | engine: copiilot
         ^^^^^^^^^
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 17/25 | Good |
| Actionability | 22/25 | Excellent |
| Context | 14/20 | Good |
| Examples | 13/15 | Excellent |
| Consistency | 9/15 | Acceptable |
| **Total** | **75/100** | **Good** |

#### Strengths
- ✅ "Did you mean: copilot?" — Levenshtein suggestion is excellent DX
- ✅ "Example: engine: copilot" — corrective example present
- ✅ Lists all valid engines
- ✅ Source context with highlighted word

#### Weaknesses
- ❌ Outer `test-2.md:1:1:` prefix is wrong — inner `17:9` is correct
- ❌ Missing documentation URL (unlike `validateEngineInlineDefinition` which links to `constants.DocsEnginesURL`)
- ❌ First thing a developer reads is the misleading `file:1:1:` prefix

</details>

<details>
<summary><b>Test Case 3: Invalid Timeout Value (`timeout-minutes: -10`)</b> — Score: 70/100 ✅</summary>

#### Test Configuration

**Workflow**: `bot-detection.md` (925 lines — complex workflow)
**Error Type**: Category C — Invalid/out-of-range value
**Error Introduced**: `timeout-minutes: -10` (minimum is 1)

#### Actual Compiler Output

```
test-3.md:1:1: error: test-3.md:826:18: error: 'timeout-minutes' (line 826, col 18): must be at least 1 (got -10). Example: timeout-minutes: 10
  826 | timeout-minutes: -10
        ^^^^^^^^^^^^^^^

Evaluation Scores

Dimension	Score	Rating
Clarity	18/25	Good
Actionability	18/25	Good
Context	14/20	Good
Examples	11/15	Good
Consistency	9/15	Acceptable
Total	70/100	Good

Strengths

• ✅ Constraint translated: "minimum: got -10, want 1" → "must be at least 1 (got -10)" — excellent plain-language translation
• ✅ "Example: timeout-minutes: 10" is helpful
• ✅ Source context highlights the field name

Weaknesses

• ❌ Same outer file:1:1: prefix issue
• ❌ No documentation link (unlike engine errors which reference constants.DocsEnginesURL)
• ❌ "Example: timeout-minutes: 10" doesn't mention the valid range (1–4320 minutes)

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	67/100
Excellent (85+)	0
Good (70–84)	2
Poor (40–54)	0
Acceptable (55–69)	1
Threshold Met (≥70)	❌ No (67 < 70)

Quality Assessment: ⚠️ Needs Improvement — Average 67/100, below the 70 threshold. The primary driver is the double-formatting bug affecting all error types.

---

Priority Improvement Recommendations

🔴 High Priority — Fix double-format bug (affects ALL errors)

Problem: isFormattedCompilerError() does not recognise pre-formatted errors returned by createFrontmatterError and validateWithSchemaAndLocation, so compiler.go wraps them again with a spurious file:1:1: error: prefix.

Location: pkg/workflow/compiler_error_formatter.go, pkg/parser/schema_compiler.go, pkg/workflow/frontmatter_error.go

Option A — Change return types (preferred, type-safe):
Return *parser.FormattedParserError from both code paths so isFormattedCompilerError recognises them:

// pkg/parser/schema_compiler.go (lines 357, 385)
// Before:
return errors.New(formattedErr)
// After:
return &FormattedParserError{formatted: formattedErr}

// pkg/workflow/frontmatter_error.go (lines 76, 80)
// Before:
return fmt.Errorf("%s\n%s", vscodeFormat, context)
// After:
return &FormattedFrontmatterError{formatted: ...}  // implement isFormattedCompilerError

Option B — String-prefix detection (simpler fallback):
In compiler.go, detect already-formatted errors by checking if err.Error() starts with the file path:

if strings.HasPrefix(err.Error(), markdownPath+":") {
    // Already formatted with file:line:col — don't double-wrap
    return &wrappedCompilerError{formatted: err.Error(), cause: err}
}

Impact: Fixes incorrect IDE navigation (file:1:1 → correct line) for ALL compile errors. Current behavior directs developers to line 1 of every file regardless of where the error actually is.

🟡 Medium Priority — Add YAML error corrective examples

Problem: YAML syntax errors show what's wrong but not what's right.

Location: pkg/workflow/frontmatter_error.go or pkg/parser/frontmatter_content.go

Suggestion: Add an example bank for common YAML errors:

// After translating the YAML message, look up a corrective example
corrections := map[string]string{
    "missing ':' after key": "Correct format: key: value",
}
if correction, ok := corrections[translatedMsg]; ok {
    message += "\n\n" + correction
}

🟢 Low Priority — Add documentation links to timeout errors

Problem: Engine errors link to constants.DocsEnginesURL but timeout errors don't.

Location: pkg/workflow/schema_validation.go (getFieldExample)

Suggestion: Add a DocsTimeoutURL constant and reference it in the timeout-minutes example:

"timeout-minutes": fmt.Sprintf("Example: timeout-minutes: 10\nValid range: 1–4320 minutes. See: %s", constants.DocsTimeoutURL),

---

Implementation Guide

Quickest fix — resolve double-wrapping in pkg/parser/schema_compiler.go:

// validateWithSchemaAndLocation — replace both returns
// Line 357:
return &parser.FormattedParserError{formatted: formattedErr}
// Line 385:
return &parser.FormattedParserError{formatted: formattedErr}

And in pkg/workflow/frontmatter_error.go, change the return type from fmt.Errorf to *parser.FormattedParserError:

return &parser.FormattedParserError{formatted: fmt.Sprintf("%s\n%s", vscodeFormat, context)}

This ensures both code paths return types that isFormattedCompilerError already knows how to handle, without changing any logic.

---

References:

• §23943220569

Generated by Daily Syntax Error Quality Check · ● 8.4M · ◷

• expires on Apr 6, 2026, 10:51 AM UTC