Fix chat response truncation and raw JSON display#635
Conversation
The previous 1024 limit was too low, causing LLM responses to be truncated mid-JSON when structured output was requested. Fixes #616.
Extract finish_reason from the OpenAI response. When it is "length", mark the result as degraded with reason "Response was truncated". Also detect invalid JSON when JSON mode was requested, which indicates the response was cut off before the model could produce valid output.
Extract finishReason from the Gemini response candidates. When it is "MAX_TOKENS", mark the result as degraded with reason "Response was truncated". Also detect invalid JSON when JSON mode was requested.
When an assistant message starts with '{' but is not valid JSON, the
frontend now shows "This response was cut short. Try a simpler question
or rephrase." instead of rendering the raw broken JSON.
Plain-text LLM responses should still fall through to the static
classifier. Only flag content that starts with '{' but fails JSON
parsing, which indicates truncated structured output.
Adversarial Self-ReviewWhat looks correct
Potential issues found
Acceptance criteria check
No issues requiring immediate fixes found. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements truncation detection for LLM responses in the Gemini and OpenAI providers by monitoring provider-specific finish reasons and validating JSON integrity. It also increases the default maximum token count and introduces a frontend notice for truncated messages. The review feedback identifies that the JSON truncation logic is currently fragile and duplicated across the backend and frontend; it is recommended to centralize this logic into a shared utility and enhance it to handle conversational prefixes or markdown formatting by using more robust brace-matching.
| internal static bool LooksLikeTruncatedJson(string text) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(text)) | ||
| return false; | ||
|
|
||
| var trimmed = text.TrimStart(); | ||
| if (!trimmed.StartsWith('{')) | ||
| return false; | ||
|
|
||
| try | ||
| { | ||
| using var doc = JsonDocument.Parse(trimmed); | ||
| return false; | ||
| } | ||
| catch (JsonException) | ||
| { | ||
| return true; | ||
| } | ||
| } |
There was a problem hiding this comment.
The current implementation of LooksLikeTruncatedJson is fragile because it assumes the response starts exactly with a {. LLMs occasionally include conversational prefixes or markdown code blocks (e.g., "Here is the JSON: ```json ...") even when instructed to return JSON.
Additionally, this logic is duplicated in OpenAiLlmProvider.cs. Consider moving this utility to a shared location like LlmInstructionExtractionPrompt and making it more robust by searching for the first and last braces, similar to the logic used in TryParseStructuredResponse.
internal static bool LooksLikeTruncatedJson(string text)
{
if (string.IsNullOrWhiteSpace(text)) return false;
var trimmed = text.Trim();
var first = trimmed.IndexOf('{');
var last = trimmed.LastIndexOf('}');
if (first >= 0 && last > first)
{
try
{
using var doc = JsonDocument.Parse(trimmed[first..(last + 1)]);
return false;
}
catch (JsonException) { }
}
return first >= 0;
}| internal static bool LooksLikeTruncatedJson(string text) | ||
| { | ||
| if (string.IsNullOrWhiteSpace(text)) | ||
| return false; | ||
|
|
||
| var trimmed = text.TrimStart(); | ||
| if (!trimmed.StartsWith('{')) | ||
| return false; | ||
|
|
||
| try | ||
| { | ||
| using var doc = JsonDocument.Parse(trimmed); | ||
| return false; | ||
| } | ||
| catch (JsonException) | ||
| { | ||
| return true; | ||
| } | ||
| } |
There was a problem hiding this comment.
This method is identical to the one in GeminiLlmProvider.cs. To improve maintainability and robustness, consider moving this logic to a shared utility class (e.g., LlmInstructionExtractionPrompt) and updating it to handle cases where the JSON is wrapped in markdown or preceded by text.
internal static bool LooksLikeTruncatedJson(string text)
{
if (string.IsNullOrWhiteSpace(text)) return false;
var trimmed = text.Trim();
var first = trimmed.IndexOf('{');
var last = trimmed.LastIndexOf('}');
if (first >= 0 && last > first)
{
try
{
using var doc = JsonDocument.Parse(trimmed[first..(last + 1)]);
return false;
}
catch (JsonException) { }
}
return first >= 0;
}| function isTruncatedJson(content: string): boolean { | ||
| if (!content) return false | ||
| const trimmed = content.trim() | ||
| if (!trimmed.startsWith('{')) return false | ||
| try { | ||
| JSON.parse(trimmed) | ||
| return false | ||
| } catch { | ||
| return true | ||
| } | ||
| } |
There was a problem hiding this comment.
The isTruncatedJson heuristic on the frontend shares the same robustness issue as the backend: it fails to detect truncation if the assistant includes any leading text or markdown formatting. Improving the brace-matching logic here will ensure the truncation notice is shown even when the response isn't a pure JSON string.
function isTruncatedJson(content: string): boolean {
const trimmed = (content || '').trim()
const first = trimmed.indexOf('{')
const last = trimmed.lastIndexOf('}')
if (first >= 0 && last > first) {
try {
JSON.parse(trimmed.substring(first, last + 1))
return false
} catch {}
}
return first >= 0
}
Cover OpenAI finish_reason=length, Gemini finishReason=MAX_TOKENS, OpenAI JSON-mode truncation, and LooksLikeTruncatedJson unit tests for both providers.
Self-review follow-up: tests addedAddressed finding #5 from the self-review (no tests for truncation paths). Added 13 new tests:
All 1942 backend tests pass. Frontend typecheck clean. |
Adversarial Code Review -- PR #635Reviewed the full diff and surrounding file context. Findings below, ordered by severity. 1. [HIGH] Frontend
|
1 participant
Summary
MaxTokensfrom 1024 to 2048 to reduce truncation frequencyfinish_reasonextraction to OpenAI provider; mark as degraded whenfinish_reason: "length"finishReasonextraction to Gemini provider; mark as degraded whenfinishReason: "MAX_TOKENS"{but fails parse) when JSON mode was requested, and mark as degraded with reason "Response was truncated"AutomationChatView.vuedetects assistant messages that look like truncated JSON and shows "This response was cut short. Try a simpler question or rephrase." instead of raw broken JSONCloses #616
Test plan
{that aren't valid JSON