Skip to content

feat: separate internal schema dimensions from embedding API request dimensions#377

Open
jszhang98 wants to merge 1 commit intoCortexReach:masterfrom
jszhang98:fix/separate-request-dimensions
Open

feat: separate internal schema dimensions from embedding API request dimensions#377
jszhang98 wants to merge 1 commit intoCortexReach:masterfrom
jszhang98:fix/separate-request-dimensions

Conversation

@jszhang98
Copy link
Copy Markdown

PR Title

feat: separate internal schema dimensions from embedding API request dimensions

Summary

This PR decouples two previously conflated dimension semantics in the embedding pipeline:

  • Internal dimensions used for LanceDB schema sizing and local vector validation.
  • Request dimensions sent to embedding providers that support variable output dimensions.

After this change:

  • embedding.dimensions is treated as internal schema/validation dimension.
  • embedding.requestDimensions is optional and only used for API request payload fields.
  • If requestDimensions is not configured, no dimensions field is sent to embedding APIs by default.
  • omitDimensions=true still has highest priority and suppresses dimensions fields even when requestDimensions is set.

Motivation

Some providers/models (for example non-matryoshka-compatible models) reject dimensions/output_dimension in request payloads.

Previously, embedding.dimensions was reused for both storage and API request parameters, which could cause provider-side 400 errors during startup/runtime.

This PR keeps storage behavior stable while preventing accidental request-parameter injection.

What Changed

  • Embedding config model extended with requestDimensions.
  • Embedder request payload dimension source changed from dimensions to requestDimensions.
  • Plugin config parsing and wiring updated to pass requestDimensions explicitly.
  • Plugin schema updated to declare embedding.requestDimensions.
  • Regression tests updated to reflect new default behavior and precedence rules.

Behavior Matrix

  • dimensions only:
    • Used internally for schema/validation.
    • Not forwarded to embedding API payload.
  • requestDimensions set:
    • Forwarded as provider-specific request field (dimensions/output_dimension).
  • requestDimensions + omitDimensions=true:
    • Request dimension fields are omitted.

Backward Compatibility

  • Existing storage behavior remains unchanged.
  • Existing users relying on implicit forwarding of embedding.dimensions to API payload should migrate to embedding.requestDimensions.
  • No change to fixed-dimension LanceDB table constraints in this PR.

Files Updated

Testing

  • Updated regression assertions for new dimension-forwarding semantics.
  • Added schema assertion for embedding.requestDimensions.
  • Local full test command was blocked in current environment due missing jiti package dependency, but modified regression logic and assertions are in place.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant