Skip to content

feat(proxy): make connection-level retries configurable per provider#148

Merged
kianwoon merged 1 commit intomainfrom
feat/configurable-connection-retries
Apr 2, 2026
Merged

feat(proxy): make connection-level retries configurable per provider#148
kianwoon merged 1 commit intomainfrom
feat/configurable-connection-retries

Conversation

@kianwoon
Copy link
Copy Markdown
Owner

@kianwoon kianwoon commented Apr 2, 2026

Summary

  • Add connectionRetries per-provider config (default 3) to control retry count before fallback escalation
  • Reduces worst-case latency from ~35s to ~8-17s for operators who set lower values
  • Backward compatible — omitting the field preserves current behavior (3 retries)

Files changed

  • src/types.ts_connectionRetries?: number on ProviderConfig
  • src/config.ts — Zod schema, peekConfig type, loadConfig mapping
  • src/proxy.tsforwardWithRetry() reads provider._connectionRetries ?? 3

Config example

providers:
  glm:
    connectionRetries: 1   # fail fast, move to fallback quickly
  minimax:
    connectionRetries: 2   # slightly more tolerant

Motivation

From 2hr production log analysis: 108 connection errors on GLM with 3 retries each wasted ~35s per failed request before fallback. With connectionRetries: 1, that drops to ~8.5s.

Test plan

  • Verify npm run build passes with no type errors
  • Set connectionRetries: 0 on a provider and confirm it skips retries, goes straight to fallback
  • Set connectionRetries: 1 and confirm exactly 1 retry before fallback
  • Omit connectionRetries and confirm default behavior (3 retries) preserved

Add `connectionRetries` config field (default 3) to control how many
times forwardWithRetry() retries a provider on TTFB/stall/connection
errors before escalating to fallback. Operators can now set low
values (1-2) to fail fast on known-bad providers, reducing worst-case
latency from ~35s to ~8-17s on connection failures.

- src/types.ts: add _connectionRetries to ProviderConfig
- src/config.ts: add to Zod schema, peekConfig type, and loadConfig mapping
- src/proxy.ts: use provider._connectionRetries ?? CONNECTION_RETRY_MAX
@kianwoon kianwoon merged commit 3b72429 into main Apr 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant