Skip to content

fix(openai): strengthen streaming tool call dedup to prevent false evictions#1510

Open
majiayu000 wants to merge 1 commit into0xPlaygrounds:mainfrom
majiayu000:fix/issue-1508-streaming-dedup
Open

fix(openai): strengthen streaming tool call dedup to prevent false evictions#1510
majiayu000 wants to merge 1 commit into0xPlaygrounds:mainfrom
majiayu000:fix/issue-1508-streaming-dedup

Conversation

@majiayu000
Copy link

Summary

The deduplication heuristic added in v0.32 (streaming.rs ~line 211) evicts a tool call entry when a new SSE chunk arrives at the same index with a different id. This breaks providers (e.g. GLM-4 via OpenAI-compatible endpoints) that send a unique id on every SSE delta chunk for the same logical tool call — each chunk triggers a false eviction, yielding incomplete fragments as "completed" tool calls.

Fix: Strengthen the eviction condition to require both id AND name to be non-empty and different from the existing entry. A genuinely new tool call at the same index (the LiteLLM/OneAPI gateway case) always carries a new non-empty name, while continuation chunks from providers like GLM-4 have an empty name.

Fixes #1508

Changes

  • Added name comparison to the dedup eviction guard: eviction now requires both a new non-empty id that differs and a new non-empty name that differs
  • Added regression test test_unique_id_per_chunk_single_tool_call reproducing the exact GLM-4 scenario from the issue

Test plan

  • New test test_unique_id_per_chunk_single_tool_call verifies GLM-4 chunks with unique IDs accumulate into one tool call
  • Existing test test_duplicate_index_different_id_tool_calls still passes (LiteLLM/OneAPI gateway case)
  • All 10 streaming tests pass
  • cargo fmt clean
  • cargo clippy clean

…ictions

The deduplication heuristic added in v0.32 evicts a tool call entry when
a new chunk arrives at the same index with a different id.  This breaks
providers (e.g. GLM-4) that send a unique id on every SSE chunk for the
same logical tool call — each chunk triggers a false eviction, yielding
incomplete fragments.

Strengthen the condition to require both id AND name to differ before
evicting.  A genuinely new tool call at the same index (the
LiteLLM/OneAPI gateway case) always carries a new non-empty name,
while continuation chunks have an empty name.

Adds a regression test reproducing the GLM-4 scenario from issue 0xPlaygrounds#1508.

Fixes 0xPlaygrounds#1508

Signed-off-by: majiayu000 <1835304752@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Streaming tool call deduplication heuristic breaks providers that send unique IDs per SSE chunk

1 participant