Feature/model agnostic converse api

[OmoleOO]

Summary

Refactors the chatbot to use AWS Bedrock's unified Converse API for model-agnostic inference, enabling support for any Bedrock model provider without code changes.

Key Changes

• Worker Lambda: Replace InvokeModelWithResponseStream with ConverseStreamCommand for unified streaming across all model providers
• Config Schema: Remove Anthropic-specific fields (anthropicVersion, topK), add modelSpecific for provider-specific parameters
• CLI Validation: Add model ID pattern matching for known providers and parameter range validation
• IAM Permissions: Add bedrock:Converse and bedrock:ConverseStream permissions

Supported Model Providers

Provider	Example Model ID
Anthropic	anthropic.claude-3-5-sonnet-20240620-v1:0
Amazon	amazon.titan-text-express-v1, amazon.nova-lite-v1:0
Meta	meta.llama3-70b-instruct-v1:0
Cohere	cohere.command-r-plus-v1:0
Mistral	mistral.mistral-large-2402-v1:0
AI21	ai21.jamba-1-5-large-v1:0
DeepSeek	deepseek.deepseek-r1-v1:0

Why Converse API?

The Converse API provides:

• Unified request/response format across all Bedrock models
• Internal handling of model-specific differences
• Streaming support via ConverseStream
• Simpler codebase without custom adapters per provider

Test Plan

• All 61 CLI tests pass (including 36 new model validation tests)
• CDK synthesizes successfully
• Manual testing with Claude model
• Manual testing with Titan model (optional)
• Verify streaming works end-to-end

Breaking Changes

Configuration schema changes require updating existing SSM parameters:

• Remove model.anthropicVersion
• Remove generation.topK
• Add modelSpecific: {} (optional)

Run npm run cli config reset to apply new defaults after deployment.

[Copilot]

Pull request overview

This PR refactors the chatbot to use the Bedrock Converse/ConverseStream APIs and introduces model-agnostic configuration and validation, so that multiple Bedrock providers can be targeted via a unified config and worker implementation.

Changes:

• CLI config validation: adds provider-aware modelId validation and generation parameter range checks, and removes Anthropic-specific schema fields.
• Worker Lambda: replaces InvokeModelWithResponseStream and custom signing logic with ConverseStreamCommand / ConverseCommand, adds support for modelSpecific fields, and introduces comprehensive unit tests around streaming, KB usage, mock mode, and inference profiles.
• Infrastructure and schema: updates the configuration schema, example config, and worker IAM to reflect Converse usage and new optional modelSpecific settings.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tools/cli/lib/config.js	Adds MODEL_PATTERNS, validateModelId, and validateGenerationParams, integrates them into validateConfig, and removes validation of model.anthropicVersion/generation.topK.
tools/cli/__tests__/model-validation.test.js	New Jest tests covering model ID validation across supported providers, generation parameter range validation, and cross-model config validation paths.
tools/cli/__tests__/config.test.js	Aligns existing config validation tests with the new schema (no anthropicVersion/topK) and ensures modelSpecific is round-tripped where appropriate.
lambda/worker/index.js	Switches the worker to Converse/ConverseStream; builds Converse request objects from the shared config, wires in optional modelSpecific via additionalModelRequestFields, and simplifies the inference-profile path using ConverseCommand.
lambda/worker/__tests__/handler.test.js	Adds unit tests for ConverseStream usage, WebSocket streaming behavior, model-specific overrides, KB retrieval integration, mock mode, and inference profile fallback.
lambda/config-schema.js	Updates the configuration schema to remove Anthropic-specific fields, document Converse-based behavior, and add the modelSpecific section to the default config.
config.example.json	Updates the example config JSON to match the new schema (no anthropicVersion/topK, adds modelSpecific).
cdk/lib/api-stack.ts	Extends the worker Lambda role with bedrock:Converse and bedrock:ConverseStream permissions while retaining legacy InvokeModel* and KB-related actions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mfittko]

✅ Approved

Excellent implementation of SSM-based dynamic configuration with a professional CLI tool.

Key Strengths:

• Clean architecture with proper separation of concerns
• Comprehensive test coverage (15 tests, all passing)
• Well-documented with IAM policies included
• Backward compatible (no breaking changes)
• Smart caching strategy (5-min TTL)

What Works Well:

• CLI tool provides intuitive config management (get, set, update, validate, backup, restore)
• Lambda worker gracefully falls back to defaults if SSM unavailable
• Knowledge Base can be enabled/disabled dynamically
• Proper error handling and validation

Minor Notes:

• Config changes take up to 5 minutes to apply (cache TTL) - acceptable and documented
• Consider adding integration tests in future PR (unit tests are comprehensive)

Ready to merge. 🚀

[mfittko]

Code Review

Found 2 issues that need attention:

Issue 1: Multi-turn conversation history dropped (HIGH SEVERITY)

The new Converse API implementation hardcodes messages to a single user message, completely ignoring job.messages which contains the conversation history. This breaks multi-turn conversations - every request is treated as isolated with no memory of prior exchanges.

Location: lambda/worker/index.js:140

Issue 2: Stream errors leave client hanging (MEDIUM SEVERITY)

When internalServerException or modelStreamErrorException occurs, the code throws without sending any WebSocket notification to the client. The frontend has no timeout or error handler, so users are left with a loading state indefinitely.

Location: lambda/worker/index.js:178-181

Checked for bugs and compliance with CLAUDE.md/AGENTS.md (no guideline files found).

[mfittko]

Multi-turn conversation history is dropped

The new code hardcodes messages to a single user message, completely ignoring job.messages which contains the conversation history.

Previous behavior: The old code (main branch, lines 143-181) normalized job.messages and included the full conversation history, allowing multi-turn conversations with context.

Impact: Every API call is now treated as a brand new conversation with no memory. Users cannot have ongoing conversations where the model remembers prior exchanges.

Suggested fix: Restore conversation history handling:

1. Parse and normalize job.messages array
2. Convert messages to Converse API format with content: [{text: '...'}] structure
3. Pass the full message history to the messages parameter

The Converse API supports multi-turn conversations - see AWS docs.

[mfittko]

Stream errors leave client hanging

When a stream error occurs (internalServerException or modelStreamErrorException), the code immediately throws without sending any notification to the WebSocket client.

Impact: The frontend has no timeout or error handler, so users are left staring at a loading/partial state indefinitely with no feedback about what went wrong.

Previous behavior: The old streaming code wrapped chunk parsing in try/catch and logged errors without throwing, allowing the stream to complete and send the complete event.

Suggested fix:
Send an error event to the client before throwing, or at minimum send a complete event to unblock the UI:

if (evt.internalServerException || evt.modelStreamErrorException) {
  const error = evt.internalServerException || evt.modelStreamErrorException
  console.log('Stream error:', error.message)
  // Notify client of error
  await ws.send(
    new PostToConnectionCommand({
      ConnectionId: connectionId,
      Data: Buffer.from(JSON.stringify({ 
        event: 'error', 
        message: error.message || 'Stream error' 
      })),
    }),
  )
  break  // Exit loop gracefully instead of throwing
}

The frontend should also be updated to handle error events.

[mfittko]

@OmoleOO can you try setting this up and then try to deploy?

https://sofatutor.slack.com/archives/CG1LWUKMJ/p1770998319357849

[mfittko]

@OmoleOO is there anything missing here? did you try the deployment?

[OmoleOO]

@OmoleOO is there anything missing here? did you try the deployment?

I need to check that nothing is missing, I'll try the deployment after.