Add user-facing message field to violation output

[oshorefueled]

Why

Terminal output showed analysis — the model's internal reasoning — as the user-facing violation message. This made findings feel verbose and opaque, as if written for an internal reviewer rather than the user.

What

• Add a message field to the LLM violation schema: required, under 15 words, written directly to the user with no rule references
• Display message instead of analysis in the terminal
• Expose analysis in native JSON output (--output=json) so both fields are available to consumers
• Add directive guidance telling the model that message is shown to the user and analysis is not

Scope

In scope

• message field added to both buildCheckLLMSchema() and buildJudgeLLMSchema() violation objects
• message added to all corresponding TypeScript violation types (JudgeLLMResult, CheckLLMResult, JudgeResult, CheckItem, CheckResult)
• Terminal display updated to use v.message
• analysis field added to the Issue interface and included in --output=json violation objects
• Directive updated to distinguish the two fields

Out of scope

• Changes to analysis field content or behavior
• Scoring, confidence, or PAT gate logic
• Vale JSON and RdJSON output formats (unaffected — they map to fixed external schemas)

Behavior impact

• Terminal output: violation messages are now sourced from message (short, user-facing) instead of analysis (internal reasoning). Users will see shorter, cleaner findings.
• --output=json: message is a new required field and analysis is now included on every violation object. Both fields are additive — existing consumers will not break.
• --output=vale-json / --output=rdjson: no change.
• Token usage: the LLM now generates one additional required string field per violation.

Risk

• Low. The terminal display change is the only behavioral difference for end users.
• JSON consumers that enumerate violation fields will see two new fields (message, analysis); both are additive.
• No scoring, config, or provider changes.

How to test / verify

Checks run

• npm run test:run — 228 tests, 32 files, all passed
• npm run build — clean build, no TypeScript errors

Manual verification

1. Run vectorlint on a file with known violations: npm run dev -- <file>
2. Confirm terminal output shows short findings (≤15 words, no rule references)
3. Run with --output=json and confirm each violation object includes both message and analysis
4. Run with --output=vale-json and confirm output is unchanged

Rollback

Revert to main. No migrations, config changes, or external dependencies introduced.

Summary by CodeRabbit

• New Features

  • Issues can include optional analysis metadata in JSON output for richer reporting.
  • Violation reports now carry a concise, user-facing message summary for clearer terminal/API output.

• Tests

  • Added tests to validate the presence and formatting (<=15 words) of the message field in violation reporting schemas.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4379f097-9bbf-4551-bac5-02d7571df2bc

📥 Commits

Reviewing files that changed from the base of the PR and between f7b4599 and 18e923e.

📒 Files selected for processing (1)

• src/cli/types.ts

---

📝 Walkthrough

Walkthrough

Adds an optional analysis field and a required user-facing message field to the violation/reporting pipeline: type definitions, LLM schemas, orchestrator reporting, and JSON output are updated so per-violation message is used for UI text and analysis is propagated as internal metadata.

Changes

Cohort / File(s)	Summary
Type definitions src/cli/types.ts, src/output/json-formatter.ts	Added analysis?: string to ReportIssueParams and Issue/JsonIssue shapes; added message?: string to violations in process params to reflect LLM payload shape.
Orchestrator / CLI src/cli/orchestrator.ts	Added reportIssue(params: ReportIssueParams) and threaded analysis through issue reporting; switched per-row text source from v.analysis → v.message when building violation output.
LLM prompts & schemas src/prompts/schema.ts, src/prompts/directive-loader.ts	Extended Check and Judge schemas so violations include a required message: string (user-facing, <=15 words) and preserved analysis as internal metadata; directive loader documents message as UI-facing.
Output formatting src/output/json-formatter.ts	JSON issue shape now optionally includes analysis when provided by the orchestrator/violations.
Tests tests/prompt-schema.test.ts	Added tests asserting message exists on check/judge violation schema items, has type string and the specified description, and that message is required.

Sequence Diagram(s)

sequenceDiagram
  participant LLM as LLM (Check/Judge)
  participant Orch as Orchestrator
  participant Formatter as JSONFormatter
  participant Out as Output (file/stdout)

  LLM-->>Orch: return violations[] { rule, match, message, analysis? }
  Orch-->>Orch: process violations (use v.message for UI row)
  Orch-->>Formatter: build JsonIssue(s) including analysis? and message
  Formatter-->>Out: emit JSON array of issues (issue.message, issue.analysis?)

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• Remove zero-violation JSON noise and restore concise LLM guidance #70: Modifies linter JSON/reporting paths and schema for analysis/message fields—strongly related to the same reporting/schema changes here.
• feat: add drop-in replacement fix field to lint output #58: Threads an optional per-violation field through reportIssue, types, JSON formatter, and schemas—matches the propagation pattern used in this change.

Poem

🐰 A hop, a nudge, a careful clue,

"Message" speaks loud, "analysis" stays true.
I nibble schema, stitch it neat,
Two fields now dance where issues meet.
Hooray — the rabbit guards the sheet! 🌿

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and concisely summarizes the main change: adding a user-facing message field to violation output, which is the core objective across schema updates, display changes, and JSON output modifications.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch enh/user-message

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}