feat:Add endpoints for factual consistency evaluation and OpenAI-compatible chat completions #53
Conversation
WalkthroughThe OpenAPI specification for the Vectara REST API v2 was updated to introduce two new endpoints: one for evaluating the factual consistency of generated text and another for OpenAI-compatible LLM chat completions, including streaming support. Several new request and response schemas were added to support these endpoints. Enhancements were made to table extraction configuration by integrating a new table generation specification. Descriptions and formatting were refined for clarity, including updates to existing endpoint documentation and schema properties. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API
%% Factual Consistency Evaluation
Client->>API: POST /v2/evaluate_factual_consistency (generated_text, source_texts)
API-->>Client: { score: 0.0 - 1.0 }
%% LLM Chat Completion
Client->>API: POST /v2/llms/chat/completions (model, messages)
alt Streaming enabled
API-->>Client: SSE stream of chat completion chunks
else Non-streaming
API-->>Client: Full chat completion response
end
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
coderabbitai
bot
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (10)
src/libs/Vectara/openapi.yaml (10)
1651-1693: Add examples and refine error cases for factual consistency endpoint
The new/v2/evaluate_factual_consistencyendpoint and its schemas align with the PR objectives. To improve usability, addx-fern-examplesfor both request and response, and consider documenting the 422 error in the components (e.g., anUnsupportedLanguageError).
4655-4672: RefineTableGenerationSpecschema for stricter validation
Currently all fields inTableGenerationSpecare optional. Consider addingminProperties: 1or markingllm_nameorprompt_templateas required to prevent empty objects. Also, explicitly definemodel_parametersshape rather than a free‐form object for better tooling support.
3949-3984: Add examples and tighten validation for factual consistency schemas
BothEvaluateFactualConsistencyRequestandEvaluateFactualConsistencyResponseare defined correctly. To improve discoverability, addexampleblocks forgenerated_text,source_texts, andscore, and enforceminItems: 1onsource_textsand a non-empty constraint ongenerated_text.
4102-4107: Enhance documentation for custom headers in OpenAI LLM requests
The newheadersproperty enables custom HTTP headers. Suggest providing an example of setting a user-agent or organization header inx-fern-examples, and clarifying allowed header names/values.
5339-5355: Encourage role enumeration inChatCompletionRequestMessage
The new message object capturesrole,content, and optionalname. For clarity, consider enumerating allowed roles (system,user,assistant, etc.) in the schema and adding a sample message underx-fern-examples.
5356-5372: Consider adding usage metrics to chat completion response
The response currently returnsobjectandchoices. To align with OpenAI’s interface, consider including an optionalusageobject (tokens used) inCreateChatCompletionResponse.
5373-5383: AugmentChatCompletionResponseChoicewith finish_reason
OpenAI’s chat API includesfinish_reasonin each choice. To support similar client behavior, consider addingfinish_reason(e.g.,stop,length) to the schema.
5394-5410: Improve streaming response schema for chat completions
CreateChatCompletionStreamResponsecorrectly defines SSE chunks, but consider adding comments about the SSE framing (data:prefix) or including an example in the spec.
5286-5286: Alignrequest_iddescription in NotFoundError schema
You’ve added a custom description forrequest_id. Ensure it matches wording inBadRequestErrorand the generalErrorschema to maintain consistency across all error types.
5497-5497: Add newline at end-of-file
YAML files should end with a single newline character to satisfy POSIX standards and tooling (YAMLLint).🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 5497-5497: no new line character at the end of file
(new-line-at-end-of-file)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (53)
src/libs/Vectara/Generated/JsonConverters.CreateChatCompletionResponseObject.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/JsonConverters.CreateChatCompletionResponseObjectNullable.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/JsonConverters.CreateChatCompletionStreamResponseObject.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/JsonConverters.CreateChatCompletionStreamResponseObjectNullable.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/JsonSerializerContext.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/JsonSerializerContextTypes.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.FactualConsistencyEvaluationClient.EvaluateFactualConsistency.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.FactualConsistencyEvaluationClient.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.IFactualConsistencyEvaluationClient.EvaluateFactualConsistency.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.IFactualConsistencyEvaluationClient.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.ILLMChatCompletionsClient.CreateChatCompletion.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.ILLMChatCompletionsClient.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.IUploadClient.UploadFile.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.IVectaraClient.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.LLMChatCompletionsClient.CreateChatCompletion.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.LLMChatCompletionsClient.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionRequestMessage.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionRequestMessage.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionResponseChoice.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionResponseChoice.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionResponseMessage.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionResponseMessage.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionStreamResponseChoice.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionStreamResponseChoice.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionStreamResponseDelta.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.ChatCompletionStreamResponseDelta.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionRequest.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionRequest.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionResponse.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionResponse.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionResponseObject.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionStreamResponse.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionStreamResponse.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateChatCompletionStreamResponseObject.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateOpenAILLMRequest.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateOpenAILLMRequestHeaders.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.CreateOpenAILLMRequestHeaders.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyRequest.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyRequest.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyRequestModelParameters.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyRequestModelParameters.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyResponse.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.EvaluateFactualConsistencyResponse.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.NotFoundError.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableExtractionConfig.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableExtractor.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableGenerationSpec.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableGenerationSpec.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableGenerationSpecModelParameters.Json.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.TableGenerationSpecModelParameters.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.Models.UploadFileRequest.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.UploadClient.UploadFile.g.csis excluded by!**/generated/**src/libs/Vectara/Generated/Vectara.VectaraClient.g.csis excluded by!**/generated/**
📒 Files selected for processing (1)
src/libs/Vectara/openapi.yaml(11 hunks)
🧰 Additional context used
🪛 YAMLlint (1.35.1)
src/libs/Vectara/openapi.yaml
[error] 5497-5497: no new line character at the end of file
(new-line-at-end-of-file)
🔇 Additional comments (6)
src/libs/Vectara/openapi.yaml (6)
560-560: Clarify optional parameters in file upload endpoint
The updated description correctly emphasizes thatmetadata,chunking_strategy, andtable_extraction_configare only applied if provided (with defaults otherwise). Consider adding a short example to illustrate default behavior when those parts are omitted.
2720-2761: Ensure consistent streaming metadata for chat completions
The/v2/llms/chat/completionsendpoint now supports streaming via SSE, but the metadata usesx-stream-based-on-body: stream()while other endpoints usestreamResponse()orstreamResponse(). Confirm that tooling and SDKs correctly interpret this pattern, or harmonize to a single convention. Also add a realistic example underx-fern-examples.
3647-3649: Approve addition ofgenerationto TableExtractionConfig
Introducing the optionalgenerationproperty toTableExtractionConfigallows downstream summarization control. This change is aligned with the newTableGenerationSpec.
4686-4688: Approve addition ofgenerationto TableExtractor
Adding the optionalgenerationreference inTableExtractoraligns with the extended table-extraction configuration. Verify SDK codegen picks up this new property.
5319-5339: Verify naming consistency for streaming flag in chat completion request
The schema usesstream(default: false), whereas other endpoints usestream_response. Ensure downstream SDKs and documentation correctly referencestream; if possible, align naming for consistency with the rest of the API.
5493-5496: Approve new OpenAPI tags for Table Extractors and chat features
The addition ofLLM Chat CompletionsandFactual Consistency Evaluationtags (along with updatingTable Extractors) properly categorizes the new endpoints.
| required: | ||
| - index | ||
| - delta | ||
| type: object | ||
| properties: | ||
| index: | ||
| type: integer | ||
| description: The index of the choice in the array of choices. | ||
| delta: | ||
| $ref: '#/components/schemas/ChatCompletionStreamResponseDelta' | ||
| description: A single chat completion chunk choice in a streaming response. |
There was a problem hiding this comment.
Critical: Missing delta.content in streaming choice schema
ChatCompletionStreamResponseChoice’s delta currently only supports role. Streaming partial messages require a content field.
Add:
properties:
delta:
$ref: '#/components/schemas/ChatCompletionStreamResponseDelta'
+ # ensure content is allowed in deltaCommittable suggestion skipped: line range outside the PR's diff.
| type: object | ||
| properties: | ||
| role: | ||
| type: string | ||
| description: 'The role of the author of this message, typically ''assistant'' for responses.' | ||
| description: A partial message update to be merged with previous chunks in a streaming response. | ||
| parameters: |
There was a problem hiding this comment.
Critical: Define content in ChatCompletionStreamResponseDelta
The delta schema only allows role. Streaming deltas need a content string property (and possibly function_call fields).
Add:
properties:
role:
type: string
+ content:
+ type: string
+ description: Partial content chunk from the assistant.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| type: object | |
| properties: | |
| role: | |
| type: string | |
| description: 'The role of the author of this message, typically ''assistant'' for responses.' | |
| description: A partial message update to be merged with previous chunks in a streaming response. | |
| parameters: | |
| type: object | |
| properties: | |
| role: | |
| type: string | |
| description: 'The role of the author of this message, typically ''assistant'' for responses.' | |
| content: | |
| type: string | |
| description: Partial content chunk from the assistant. | |
| description: A partial message update to be merged with previous chunks in a streaming response. | |
| parameters: |
| ChatCompletionResponseMessage: | ||
| required: | ||
| - role | ||
| type: object | ||
| properties: | ||
| role: | ||
| type: string | ||
| description: 'The role of the author of this message, typically ''assistant'' for responses.' | ||
| description: A message in the chat completion response from the assistant. | ||
| CreateChatCompletionStreamResponse: |
There was a problem hiding this comment.
Critical: Missing content in response message schema
ChatCompletionResponseMessage only defines role but omits the required content property, which will break deserialization of assistant replies.
Add:
properties:
role:
type: string
- # missing content
+ content:
+ type: string
+ description: The content of the assistant’s message.Committable suggestion skipped: line range outside the PR's diff.
Summary by CodeRabbit
New Features
Improvements