Remove zero-violation JSON noise and restore concise LLM guidance by oshorefueled · Pull Request #70 · TRocket-Labs/vectorlint

oshorefueled · 2026-03-04T13:49:00Z

Summary

This PR removes synthetic JSON findings that were emitted when no
violations were actually surfaced after PAT filtering. It also adds
explicit schema guidance for analysis and suggestion fields so
structured LLM output stays concise. Regression tests were added for
both behavior paths.

Why

JSON consumers were receiving placeholder findings on clean results,
and generated violation text had become too verbose for UI usage. This
change restores predictable machine-readable output and concise feedback.

Scope

In scope:
- Stop emitting fallback JSON and Vale JSON issues when surfaced
  violation count is zero.
- Add concise description constraints for analysis and suggestion
  in check and judge schemas.
- Add regression tests for the JSON-noise path and schema constraints.
Out of scope:
- Changes to scoring logic, confidence threshold behavior, or PAT gate
  decision logic.
- Output contract changes for real surfaced violations.
- Existing unrelated workspace changes (package-lock.json,
  plan-fix-noisy-verbose-json.md).

Behavior Impact

User-facing changes: yes.
- --output=json and --output=vale-json no longer include dummy
  entries like "No issues found" when no violations are surfaced.
- Real surfaced violations still include the message field as before.
Breaking changes: no.
Operational impact: none (no migration, config, or deployment changes).

Risk and Mitigations

Risk level: low.
Primary risks:
- Consumers that implicitly relied on placeholder issue rows may see
  fewer issue objects.
- Schema description changes may slightly alter model phrasing.
Mitigations:
- Added regression tests for zero-violation JSON and Vale JSON output.
- Added dedicated schema tests asserting analysis/suggestion
  descriptions for both check and judge schemas.
Rollback plan:
- Revert commit 79ad8df to restore previous emission behavior and
  schema field descriptions.

API / Contract / Schema Changes

Behavioral contract adjustment:
- For zero surfaced violations, JSON/Vale JSON now emit no synthetic
  issue object.
Structured schema metadata update:
- analysis description: "A concise 1-2 sentence explanation of the
  specific issue."
- suggestion description: "Suggest a fix in 15 words or less."

How to test / verify

Checks run

npm run test:run -- tests/orchestrator-filtering.test.ts tests/prompt-schema.test.ts
- Passed (2 files, 7 tests).
npm run test:run
- Passed (32 files, 223 tests).

Manual verification

Run vectorlint on a file with no surfaced violations using
--output=json and --output=vale-json.
Confirm output contains no placeholder issue with message
"No issues found".
Run on a file with known surfaced violations and confirm issue objects
still include a message field.

Follow-ups

Optional: add an integration fixture test that snapshots full CLI JSON
output for a no-violation run.

Known Tradeoffs

Prompt concision guidance uses schema descriptions rather than hard
token limits, so wording tightness remains model-dependent.

Summary by CodeRabbit

Bug Fixes
- JSON and Vale JSON outputs no longer produce extraneous messages when no violations are detected, resulting in cleaner output.
Tests
- Added tests to validate JSON output filtering behavior and schema field constraint descriptions.

- Stop emitting synthetic JSON/vale-json issues when filtered violations leave no surfaced findings, so clean runs no longer include noise cards. - Add explicit analysis/suggestion descriptions to judge/check structured output schemas to keep model responses concise and actionable. - Add regression tests for zero-violation JSON paths and schema description constraints to prevent future drift.

coderabbitai · 2026-03-04T13:49:11Z

Caution

Review failed

Pull request was closed or merged during review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 93780db7-0a67-4711-86db-74bbef3d2a45

📥 Commits

Reviewing files that changed from the base of the PR and between 581751f and 5fecbc3.

⛔ Files ignored due to path filters (1)

package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (4)

src/cli/orchestrator.ts
src/prompts/schema.ts
tests/orchestrator-filtering.test.ts
tests/prompt-schema.test.ts

💤 Files with no reviewable changes (1)

src/cli/orchestrator.ts

📝 Walkthrough

Walkthrough

The PR removes conditional logic in the orchestrator that automatically reports summary messages in JSON outputs when no violations are surfaced, and adds descriptive metadata to analysis and suggestion fields in the prompt schemas to guide LLM output constraints. Test coverage is added for both changes.

Changes

Cohort / File(s)	Summary
Orchestrator Reporting `src/cli/orchestrator.ts`	Removes conditional block that emitted a JSON/ValeJson report when no surfaced violations existed but a scored message was available, eliminating automatic summary message reporting in these output formats.
Prompt Schema Metadata `src/prompts/schema.ts`	Extends `buildJudgeLLMSchema` and `buildCheckLLMSchema` to annotate `analysis` and `suggestion` fields with type declarations and descriptive constraints (concise 1-2 sentence explanations and fixes in ≤15 words).
Orchestrator Tests `tests/orchestrator-filtering.test.ts`	Adds two new tests verifying that JSON and Vale JSON outputs contain zero issues and exclude dummy violations when only low-confidence violations are present.
Schema Tests `tests/prompt-schema.test.ts`	Adds unit tests validating that both judge and check schema definitions properly expose the new analysis and suggestion field descriptors with expected type and description metadata.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

Only score based on surfaced violations only #69 — Directly related; modifies the same orchestrator non-judge/check reporting flow for handling no surfaced violations in JSON/ValeJson outputs.
Rename rule types from semi-objective/subjective to check/judge #49 — Modifies src/prompts/schema.ts schema definitions and type discriminators affecting the same analysis/suggestion field structure.
Align CLI surfacing with PAT filters and expose confidence tuning #66 — Related; alters orchestrator violation counting and JSON output reporting behavior.

Suggested reviewers

ayo6706

Poem

🐰 Hop hooray! The schemas speak more clear,
With hints for analysis bright and dear,
No false reports when violations flee,
Just honest tests for you and me!
Better prompts in fifteen words—what glee! ✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main changes: removing JSON output noise for zero-violation cases and adding LLM schema constraints for concise guidance.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings (stacked PR)
📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch vcli-fix-noise

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

- Remove an unused check-result variable left after JSON-noise cleanup. - Parse mocked console output inline in tests to avoid unsafe-any assignments while preserving assertions.

- Replace inline parsed-output shapes with Result and ValeOutput types to simplify test code and keep contract checks explicit. - Preserve existing no-dummy-issue assertions and runtime behavior.

- Keep main's surfaced-violation scoring flow in orchestrator so check severity and score derive from filtered violations plus word count. - Keep branch behavior that removes synthetic JSON/Vale JSON issues when no violations are surfaced, preventing noisy empty findings. - Update filtering tests to RawCheckResult contracts while retaining zero-violation JSON regression assertions and typed output parsing.

oshorefueled added 2 commits March 4, 2026 14:40

chore: sync package-lock to 2.4.0

8fe23df

oshorefueled added 3 commits March 4, 2026 14:51

refactor(cli): satisfy lint in filtering path

c58729b

- Remove an unused check-result variable left after JSON-noise cleanup. - Parse mocked console output inline in tests to avoid unsafe-any assignments while preserving assertions.

test(cli): use concrete output types in JSON assertions

2f233a9

- Replace inline parsed-output shapes with Result and ValeOutput types to simplify test code and keep contract checks explicit. - Preserve existing no-dummy-issue assertions and runtime behavior.

oshorefueled merged commit e5cefab into main Mar 4, 2026
2 of 3 checks passed

coderabbitai bot mentioned this pull request Mar 4, 2026

Add user-facing message field to violation output #71

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove zero-violation JSON noise and restore concise LLM guidance#70

Remove zero-violation JSON noise and restore concise LLM guidance#70
oshorefueled merged 5 commits intomainfrom
vcli-fix-noise

oshorefueled commented Mar 4, 2026 •

edited by coderabbitai bot

Loading

Uh oh!

coderabbitai bot commented Mar 4, 2026 •

edited

Loading

Review failed

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

oshorefueled commented Mar 4, 2026 • edited by coderabbitai bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Scope

Behavior Impact

Risk and Mitigations

API / Contract / Schema Changes

How to test / verify

Follow-ups

Known Tradeoffs

Summary by CodeRabbit

Uh oh!

coderabbitai bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review failed

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Poem

❌ Failed checks (1 warning)

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

oshorefueled commented Mar 4, 2026 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Mar 4, 2026 •

edited

Loading