Skip to content

fix: make upstream timeout configurable, default to 300s #799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
syedhashmi wants to merge 6 commits into
base: main
Choose a base branch
from
+11 −11
Open

fix: make upstream timeout configurable, default to 300s #799

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
0c7b999
fix: make upstream timeout configurable, default to 300s
Mar 3, 2026
00bd110
fix: make upstream timeout configurable and increase default to 300s …
Mar 5, 2026
3e18289
use per-listener timeout config instead of separate upstream_timeout …
Mar 16, 2026
125af57
remove upstream_timeout_ms from config, schema, and rust struct
Mar 16, 2026
95486db
revert prompt_gateway changes
Mar 16, 2026
8cb3471
fix: use prompt_gateway_listener.timeout for outbound_api_traffic lis…
Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cli/planoai/utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ def convert_legacy_listeners(
"type": "model_listener",
"port": 12000,
"address": "0.0.0.0",
"timeout": "30s",
"timeout": "300s",
"model_providers": model_providers or [],
}

Expand All @@ -101,7 +101,7 @@ def convert_legacy_listeners(
"type": "prompt_listener",
"port": 10000,
"address": "0.0.0.0",
"timeout": "30s",
"timeout": "300s",
}

# Handle None case
Expand Down
10 changes: 5 additions & 5 deletions config/envoy.template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -257,7 +257,7 @@ static_resources:
route:
auto_host_rewrite: true
cluster: bright_staff
timeout: 300s
timeout: {{ prompt_gateway_listener.timeout | default('300s') }}
{% for cluster_name, cluster in plano_clusters.items() %}
- match:
prefix: "/"
Expand All @@ -268,7 +268,7 @@ static_resources:
route:
auto_host_rewrite: true
cluster: {{ cluster_name }}
timeout: 300s
timeout: {{ prompt_gateway_listener.timeout | default('300s') }}
{% endfor %}
http_filters:
- name: envoy.filters.http.router
Expand Down Expand Up @@ -336,7 +336,7 @@ static_resources:
auto_host_rewrite: true
prefix_rewrite: "/agents/"
cluster: bright_staff
timeout: {{ listener.timeout | default('30s') }}
timeout: {{ listener.timeout | default('300s') }}
http_filters:
- name: envoy.filters.http.compressor
typed_config:
Expand Down Expand Up @@ -517,12 +517,12 @@ static_resources:
route:
auto_host_rewrite: true
cluster: {{ llm_cluster_name }}
timeout: 300s
timeout: {{ llm_gateway_listener.timeout }}
{% if llm_gateway_listener.max_retries %}
retry_policy:
retry_on: "5xx,connect-failure,refused-stream,reset,retriable-status-codes"
num_retries: {{ llm_gateway_listener.max_retries }}
per_try_timeout: 30s
per_try_timeout: {{ llm_gateway_listener.timeout }}
retriable_status_codes: [429, 500, 502, 503, 504]
retry_back_off:
base_interval: 0.5s
Expand Down
8 changes: 4 additions & 4 deletions crates/common/src/consts.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@ pub const SYSTEM_ROLE: &str = "system";
pub const USER_ROLE: &str = "user";
pub const TOOL_ROLE: &str = "tool";
pub const ASSISTANT_ROLE: &str = "assistant";
pub const ARCH_FC_REQUEST_TIMEOUT_MS: u64 = 30000; // 30 seconds
pub const DEFAULT_TARGET_REQUEST_TIMEOUT_MS: u64 = 30000; // 30 seconds
pub const API_REQUEST_TIMEOUT_MS: u64 = 30000; // 30 seconds
pub const MODEL_SERVER_REQUEST_TIMEOUT_MS: u64 = 30000; // 30 seconds
pub const ARCH_FC_REQUEST_TIMEOUT_MS: u64 = 300_000; // 300 seconds
pub const DEFAULT_TARGET_REQUEST_TIMEOUT_MS: u64 = 300_000; // 300 seconds
pub const API_REQUEST_TIMEOUT_MS: u64 = 300_000; // 300 seconds
pub const MODEL_SERVER_REQUEST_TIMEOUT_MS: u64 = 300_000; // 300 seconds
pub const MODEL_SERVER_NAME: &str = "bright_staff";
pub const ARCH_ROUTING_HEADER: &str = "x-arch-llm-provider";
pub const MESSAGES_KEY: &str = "messages";
Expand Down
Loading