Add LLM-assisted instruction extraction from natural language chat#586
Add LLM-assisted instruction extraction from natural language chat#586Chris0Jeky merged 12 commits intomainfrom
Conversation
Extend LlmCompletionResult with optional Instructions list for LLM-extracted structured instructions. Extend ChatCompletionRequest with optional SystemPrompt field for provider-specific system prompts.
Shared system prompt and JSON response parser used by OpenAI and Gemini providers to extract actionable board instructions from natural language chat messages. Handles markdown code fences, missing fields, and malformed JSON gracefully.
Add system prompt for instruction extraction, request JSON mode via response_format, and parse structured output into LlmCompletionResult.Instructions. Falls back to static LlmIntentClassifier when structured parse fails.
Add system prompt prepended as user message, request JSON via responseMimeType, and parse structured output into LlmCompletionResult.Instructions. Falls back to static LlmIntentClassifier when structured parse fails.
When LlmCompletionResult.Instructions has entries, iterate each and call ParseInstructionAsync individually. Falls back to raw user message when no instructions are extracted. Supports multiple proposals from a single message.
OpenAI and Gemini providers now prepend a system prompt message, so role mapping tests expect two messages instead of one.
LlmInstructionExtractionPromptTests: 12 tests covering valid/invalid JSON, code fences, missing fields, multiple instructions, and system prompt content. ChatServiceTests: 4 new tests for LLM-extracted instruction flow, fallback behavior, multi-instruction proposals, and empty instructions list handling.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request implements LLM-assisted instruction extraction for the OpenAI and Gemini providers, enabling the conversion of natural language chat messages into structured board actions. Key changes include a new shared system prompt, JSON response parsing logic, and updates to the ChatService to support multiple extracted instructions. Feedback identifies a high-severity issue in the Gemini provider where system prompts are incorrectly prepended as user messages, which may violate API role requirements. Furthermore, the markdown stripping logic for parsing LLM responses is noted as fragile and should be improved using a more robust brace-matching strategy.
| var systemPrompt = request.SystemPrompt ?? LlmInstructionExtractionPrompt.SystemPrompt; | ||
| var allMessages = new List<object> | ||
| { | ||
| new | ||
| { | ||
| role = "user", | ||
| parts = new[] { new { text = systemPrompt } } | ||
| } | ||
| }; | ||
| allMessages.AddRange(request.Messages.Select(MapMessage)); | ||
|
|
||
| message.Content = JsonContent.Create(new | ||
| { | ||
| contents = request.Messages.Select(MapMessage).ToArray(), | ||
| contents = allMessages.ToArray(), | ||
| generationConfig = new | ||
| { | ||
| temperature = request.Temperature, | ||
| maxOutputTokens = request.MaxTokens | ||
| maxOutputTokens = request.MaxTokens, | ||
| responseMimeType = "application/json" | ||
| } | ||
| }); |
There was a problem hiding this comment.
The current implementation prepends the system prompt as a user message. This will cause API errors for any multi-turn conversation, as the Gemini API requires roles to alternate between user and model. Prepending a user message will result in two consecutive user messages at the start of the conversation.
The recommended way to provide system instructions to Gemini models is by using the system_instruction field at the top level of the request payload. This avoids altering the message history and ensures the role sequence remains valid.
var systemPrompt = request.SystemPrompt ?? LlmInstructionExtractionPrompt.SystemPrompt;
message.Content = JsonContent.Create(new
{
contents = request.Messages.Select(MapMessage).ToArray(),
system_instruction = new
{
parts = new[] { new { text = systemPrompt } }
},
generationConfig = new
{
temperature = request.Temperature,
maxOutputTokens = request.MaxTokens,
responseMimeType = "application/json"
}
});| var trimmed = responseBody.Trim(); | ||
| if (trimmed.StartsWith("```", StringComparison.Ordinal)) | ||
| { | ||
| var firstNewline = trimmed.IndexOf('\n'); | ||
| if (firstNewline >= 0) | ||
| trimmed = trimmed[(firstNewline + 1)..]; | ||
| if (trimmed.EndsWith("```", StringComparison.Ordinal)) | ||
| trimmed = trimmed[..^3].TrimEnd(); | ||
| } |
There was a problem hiding this comment.
The current logic for stripping markdown code fences is not fully robust. It can fail if the LLM returns a fenced JSON block without a newline after the language specifier (e.g., ```json{...}```). This would cause JSON parsing to fail and the system to incorrectly fall back to the static classifier.
A more resilient approach is to find the first opening brace { and the last closing brace } to extract the JSON object, as this is less sensitive to variations in markdown formatting.
var firstBrace = responseBody.IndexOf('{');
var lastBrace = responseBody.LastIndexOf('}');
if (firstBrace == -1 || lastBrace < firstBrace)
return false; // Not a valid JSON object structure
var trimmed = responseBody[firstBrace..(lastBrace + 1)];Probe requests now pass SystemPrompt = "" to avoid forcing JSON mode, which would cause the LLM to return JSON instead of the expected "OK" response. Instruction extraction system prompt and JSON mode are only applied when SystemPrompt is null (the default for chat requests).
Adversarial Self-ReviewIssues Found and Fixed
Remaining ConsiderationsPrompt injection risk (low):
Edge cases handled:
Potential follow-up items:
|
…pt as user message The Gemini API supports a top-level system_instruction field for system prompts. Sending the system prompt as a user message breaks multi-turn conversations by creating consecutive user messages. This moves the system prompt to system_instruction and omits it when empty (e.g. for probe requests). Updates tests to verify the new payload structure.
…raction
The regex-based code fence stripping fails when the LLM returns JSON
without a newline after the language specifier (e.g. ```json{...}```).
Use brace-matching (first '{' to last '}') to reliably extract the JSON
object regardless of surrounding text or formatting. Adds tests for the
no-newline edge case, bare code fences, and JSON with surrounding prose.
Gemini code review findings fixed1. System prompt sent via
|
1 participant
Summary
LlmInstructionExtractionPromptdefines the system prompt and response parserCloses #573
Test plan
LlmInstructionExtractionPromptTests(12 tests) verify JSON parsing, code fences, edge casesChatServiceTests(4 tests) verify instruction extraction flow, fallback, multi-proposaldotnet testpasses clean across all projects (1,544 total tests)