Skip to content

feat(embedding): add litellm as embedding provider#853

Open
stubbi wants to merge 1 commit intovolcengine:mainfrom
stubbi:feat/litellm-embedding-provider
Open

feat(embedding): add litellm as embedding provider#853
stubbi wants to merge 1 commit intovolcengine:mainfrom
stubbi:feat/litellm-embedding-provider

Conversation

@stubbi
Copy link

@stubbi stubbi commented Mar 21, 2026

Summary

Adds LiteLLM as a new embedding provider, resolving the gap between VLM (which already supports litellm) and the embedding layer. This enables users to route embedding requests through OpenRouter, Ollama, vLLM, and any OpenAI-compatible endpoint via litellm's unified interface.

  • New file: openviking/models/embedder/litellm_embedders.pyLiteLLMDenseEmbedder class extending DenseEmbedderBase
  • Updated: embedding_config.py — added "litellm" to the provider validation list, factory registry, and docstrings
  • Updated: embedder/__init__.py — added conditional import (graceful fallback if litellm not installed)
  • New tests: tests/unit/test_litellm_embedder.py — 15 tests covering embed, batch embed, non-symmetric mode, factory integration, and config validation

Usage example

"embedding": {
  "dense": {
    "provider": "litellm",
    "model": "openai/text-embedding-3-small",
    "api_base": "https://openrouter.ai/api/v1",
    "api_key": "<openrouter-key>",
    "dimension": 1536
  }
}

Closes #847

Test plan

  • All 15 new unit tests pass
  • All existing embedding tests still pass (no regressions)
  • Manual test with OpenRouter endpoint
  • Manual test with local Ollama embeddings

🤖 Generated with Claude Code

Adds LiteLLM as a new embedding provider, bringing embedding parity with
the VLM layer which already supports litellm. This enables users to route
embedding requests through OpenRouter, Ollama, vLLM, and any other
OpenAI-compatible endpoint via litellm's unified interface.

Closes volcengine#847

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions
Copy link

Failed to generate code suggestions for PR

Copy link
Collaborator

@ZaynJarvis ZaynJarvis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Critical Issues Found

1. Module-level side effect on import (litellm_embedders.py:13)

os.environ.setdefault("LITELLM_LOCAL_MODEL_COST_MAP", "True")

This mutates the process environment at import time — not when the embedder is used. Since __init__.py imports this module (behind try/except), this fires as soon as the embedder package is loaded, even if the user never uses litellm. This can interfere with other litellm usage in the same process that intentionally sets this env var differently.

Fix: Move this inside LiteLLMDenseEmbedder.__init__ or into _build_kwargs.

2. Probe API call during __init__ / silent fallback (litellm_embedders.py:89-93)

def _detect_dimension(self) -> int:
    try:
        result = self.embed("test")
        return len(result.dense_vector) if result.dense_vector else 1536
    except Exception:
        return 1536

When dimension=None, the constructor calls _detect_dimension() which calls self.embed("test"). This means:

  1. A billable API call the user didn't ask for, fired during object construction
  2. A network side effect inside __init__ — surprising and hard to test
  3. Silent fallback to 1536 on any exception (auth failure, network error, wrong model) — the user gets no indication their config is wrong

Fix: Require dimension as a mandatory parameter (other providers do this), or at minimum make the probe call lazy and log a warning on fallback instead of silently swallowing errors.

3. Missing None guard for LiteLLMDenseEmbedder in factory

The factory in embedding_config.py imports LiteLLMDenseEmbedder but doesn't check if it's None (which happens when litellm isn't installed). Other providers like Gemini have this guard. If a user configures provider: "litellm" without litellm installed, they'll get a confusing TypeError: 'NoneType' object is not callable instead of a clear error message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

Support litellm (or OpenRouter) as embedding provider

3 participants