Skip to content

add top-level routing_preferences with selection_policy and model metrics fetch#849

Open
adilhafeez wants to merge 3 commits intomainfrom
adil/top-level-routing-preferences
Open

add top-level routing_preferences with selection_policy and model metrics fetch#849
adilhafeez wants to merge 3 commits intomainfrom
adil/top-level-routing-preferences

Conversation

@adilhafeez
Copy link
Copy Markdown
Contributor

Closes #848

Summary

  • Adds top-level routing_preferences config format (v0.4.0+) where each route groups a pool of candidate models with a selection_policy (cheapest, fastest, random)
  • Adds model_metrics_sources — a single HTTP endpoint returning {"cost": {...}, "latency": {...}} for all models, with optional background refresh
  • Inline routing_preferences in request body overrides config-defined preferences per-request
  • Version enforcement: v0.4.0 forbids per-provider routing_preferences in model_providers; startup validates all referenced models exist

Config example

version: v0.4.0

model_providers:
  - model: openai/gpt-4o
    access_key: $OPENAI_API_KEY
  - model: openai/gpt-4o-mini
    access_key: $OPENAI_API_KEY
    default: true

routing_preferences:
  - name: code understanding
    description: understand and explain existing code snippets
    models:
      - openai/gpt-4o
      - openai/gpt-4o-mini
    selection_policy:
      prefer: cheapest

model_metrics_sources:
  url: https://example.com/model-metrics
  refresh_interval: 300
  # Response: { "cost": {"openai/gpt-4o": 0.005, ...}, "latency": {"openai/gpt-4o": 200, ...} }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: top-level routing_preferences with selection_policy and metrics fetch (v0.4.0)

1 participant