Skip to content

Add external HTTP routing policy provider with policy_id lookup and TTL cache#817

Draft
Spherrrical wants to merge 2 commits intomainfrom
musa/813
Draft

Add external HTTP routing policy provider with policy_id lookup and TTL cache#817
Spherrrical wants to merge 2 commits intomainfrom
musa/813

Conversation

@Spherrrical
Copy link
Copy Markdown
Collaborator

@Spherrrical Spherrrical commented Mar 11, 2026

Summary

  • Add support for resolving routing preferences from a configured external HTTP policy provider when a request includes policy_id and no inline routing_policy.
  • Implement in-memory TTL caching keyed by policy_id, with hybrid failure handling: fallback on transient provider errors, return request errors on invalid payloads (including policy_id mismatch).
  • Extend config/runtime/docs to support routing.policy_provider (url, headers, ttl_seconds) and preserve routing precedence: inline policy > external policy_id policy > metadata policy > config defaults.

Key Changes

  • Runtime
    • Added PolicyProviderClient for external fetch/validation.
    • Added PolicyCache for TTL-based local policy caching.
    • Wired policy_id extraction and policy resolution into routing flow (/v1/* and /routing/* paths).
  • Config
    • Added routing.policy_provider to Rust config and schema.
    • Added semantic validation for provider URL/header/TTL in CLI config generator.
    • Added backward-compatible alias for routing.llm_provider -> routing.model_provider.
  • Docs
    • Documented external policy provider configuration, request format with policy_id, expected provider payload, and resolution order.

Behavior

  • Fetch request: GET <policy_provider.url>?policy_id=<id>
  • Cache key: policy_id
  • Cache validity: ttl_seconds (default 300s when unspecified)
  • Error semantics:
    • Transient provider failures (network/timeout/5xx) -> fallback to next source
    • Invalid provider response / policy_id mismatch / non-success 4xx -> return request error

Test Plan

  • cargo test -p brightstaff --lib
  • Verified targeted tests for:
    • cache hit/miss and TTL expiry
    • provider fetch and headers
    • fallback on transient provider errors
    • request error on invalid payload / policy_id mismatch / provider client-error status

fixes #812

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support loading routing policy from external HTTP endpoint

1 participant