Qwen3 MoE Tool Usage Failures

[sempervictus]

Seeing issues such as antinomyhq/forge#1354 all related to tool use at this point (now that #261 is in). Qwen3 Coder should not be failing file operations all of the time trying to write code to actual files and i'm now seeing the same failure (agent/app believes it called a tool but gets unexpected results or none at all) modality from several distinct client types. They use a specialized tool template according to their docs - is something additional needed to leverage that?

[guoqingbao]

Seeing issues such as antinomyhq/forge#1354 all related to tool use at this point (now that #261 is in). Qwen3 Coder should not be failing file operations all of the time trying to write code to actual files and i'm now seeing the same failure (agent/app believes it called a tool but gets unexpected results or none at all) modality from several distinct client types. They use a specialized tool template according to their docs - is something additional needed to leverage that?

I'm not sure is that related to chunked prefill? It should not change the result, but I restored repetition penalty, you may change that to 1.0 （no penalty）.

[sempervictus]

it started before that so i dont think its the prefill, its also not consistent - observe kilocode succeeding at a file read and failing at a file write when it had succeeded at performing a file write a couple of minutes prior in the same execution thread:

Now I can see the exact context. I need to update the filter_prefill_finished function in src/scheduler/mod.rs to use the configurable prefill_chunk_size instead of the hardcoded CHUNK_SIZE. Let me make this change:

<update_file>
<args>
<file>
<path>src/scheduler/mod.rs</path>

</file> <content> use crate::scheduler::{block_engine::AllocStatus, sequence::SequenceStatus};
use self::{block_engine::BlockEngine, cache_engine::CacheConfig, sequence::SequenceGroup};
 ...
    self.swapped_out.make_contiguous().reverse();
}

txt



}
</content>
</update_file>

forge seems to be the most consistent reproducer of the issue

[guoqingbao]

it started before that so i dont think its the prefill, its also not consistent - observe kilocode succeeding at a file read and failing at a file write when it had succeeded at performing a file write a couple of minutes prior in the same execution thread:

Now I can see the exact context. I need to update the filter_prefill_finished function in src/scheduler/mod.rs to use the configurable prefill_chunk_size instead of the hardcoded CHUNK_SIZE. Let me make this change:

<update_file>


src/scheduler/mod.rs

use crate::scheduler::{block_engine::AllocStatus, sequence::SequenceStatus};
use self::{block_engine::BlockEngine, cache_engine::CacheConfig, sequence::SequenceGroup};
...
self.swapped_out.make_contiguous().reverse();
}

txt

}

</update_file>
forge seems to be the most consistent reproducer of the issue

Is that issue occured before? at what revision it is workable? For isq model, the model accuracy may not as high as the original one, you may try loading safetensor model without isq (although that might be slower in decoding since no fused moe used).

[sempervictus]

@guoqingbao: The problem started right around when i filed that forge issue (within the last week) - i was running at q4k without seeing this issue but have been using q8_0 mostly on the 4-way systems which should be more precise than the 4bit but exhibits issues like this:

There's also a 2-way V100 splitting a q4k Q3Coder and it exhibits the same exact problem. Did the updated quantization process add any padding or additional rounding which could explain the occasional reproduction of the tool problem?

[sempervictus]

@guoqingbao - looks like there's another way to force bad behavior: even with --penalty 1.1 --isq q8_0 --prefill-chunk-size 32768 the model can be induced into hallucinatory behaviors by passing a large amount of text:

explain what this diff does: "$(head -1000 large-diff.patch)"

557777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777Error: Aborted.

here's what happens when i hit the same model at --isq q4k split between two V100s (same architecture, different quantization):

Looking at the provided Rust code, I'll analyze the main components of the AI chat application, focusing on the key features and structure. Here's a comprehensive analysis:

## Overview

This is a comprehensive AI chat application written in Rust with extensive support for various AI providers, including OpenAI, Claude, and others. The code is organized into multiple modules handling different aspects of the application.

## Key Features

### 1. **Multi-Provider Support**
The application supports multiple AI providers through a modular client system:
...

something appears to be screwy in the 8b quantization process but i have seen the tool occasional nonsense/tool problem on the 2-way q4k as well.

[sempervictus]

So far have been able to reproduce with q8_0 pretty consistently, q4k occasionally, q5_1 not as of yet - it is quite slow however, especially in prefill - from ~60t/s down to ~30, q4_1 is also not throwing errors yet but ~40T/s with prefill rate down from ~850 -> 650T/s. If its all internally q8_1 anyway - is it possible to load the model that way?

q6k on a 4-way produces:

[00:00:13] ##----------------------------------------------------------    1/48   On Rank 0 Device
candle-vllm-swe  | [00:00:13] ##----------------------------------------------------------    1/48   On Rank 1 Device
candle-vllm-swe  | [00:00:13] ##----------------------------------------------------------    1/48   On Rank 2 Device
candle-vllm-swe  | [00:00:00] ------------------------------------------------------------    0/48   On Rank 3 Device                                                              2025-08-17 03:35:59 WARN candle_vllm::backend::heartbeat Err(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })
candle-vllm-swe  | 2025-08-17 03:35:59 WARN candle_vllm::backend::heartbeat heartbeat detection failed, exit the current process because of Err(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" })

instead of the usual fitment error

[sempervictus]

@guoqingbao :
q4_1, q4_0, and q5_1 are all susceptible to the large input bug:

$ aichat explain what this diff does: "$(head -1000 large-diff.patch)"
4033874943740755978494570007102433597777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777Error: Aborted.

it can actually register and error but its not a crash and no stack trace present:

candle-vllm-swe  | 2025-08-17 10:36:40 WARN candle_vllm::openai::pipelines::llm_engine New Request with length 11890 tokens (cmpl-192d439b-7fec-49ea-9441-665fc8313b2e).
candle-vllm-swe  | 2025-08-17 10:36:40 WARN candle_vllm::openai::pipelines::llm_engine Sending 1 tasks to 3 subprocesses
candle-vllm-swe  | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Send stream response error! (sequence id 2)
candle-vllm-swe  | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Decoding 279 tokens finished in 7 seconds (cmpl-192d439b-7fec-49ea-9441-665fc8313b2e)
candle-vllm-swe  | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Sending abort message (1 sequence(s)) to subprocesses!
candle-vllm-swe  | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine engine.write write for aborted_sequences

which seems to be related to setting the prefill chunk size (32k is what i'm trying to set as the lower threshold) - increasing the prefill size appears to make the problem above far more likely to occur. Might be useful to have the option of disabling prefill chunking (fallback to original behavior) by setting it to zero; at least to help narrow down localization of concerns like this.

[sempervictus]

The default --repeat penalty might not be enough :-\. Went into this happy loop when i told it to add q8_1 to the available quantizations (i see it implemented in Candle):

FORGE candle-vllm master                                                                                                                                                                                                                 [0.1.0/Qwen3-Coder-30B-A3B-Instruct/~0]
❯ /retry
<forge_tool_fs_read path="src/openai/models/mod.rs" />
<forge_tool_fs_read path="src/openai/mod.rs" />
<forge_tool_fs_read path="src/lib.rs" />
<forge_tool_fs_read path="Cargo.toml" />
...
<forge_tool_read path="Cargo.toml" />
<forge_tool_read path="src/openai/models/linear.rs" />
<forge_tool_read path="src/openai/models/qwen3_moe.rs" />
<forge_tool_read path="src/openai/models/mod.rs" />
<forge_tool_read path="src/openai/mod.rs" />
<forge_tool_read path="src/lib.rs" />
<forge_tool_read path="Cargo.toml" />
<forge_tool_read path="src/openai/models/linear.rs" />
<forge_tool_read path="src/openai/models/qwen3_moe.rs" />
<forge_tool_read path="src/openai/models/mod.rs" />
<forge_tool_read path="src/openai/mod.rs" />
<forge_tool_read path="
FORGE candle-vllm master 

^^ maxed out GPUs after that incomplete line at the end but nothing in the candle-vllm output indicating an error. Taking another shot at --penalty 1.2.
I've a separate kline-code task running against the repo using another 4x32Gv100 set with q4_1 which hasnt blown up in the last 30m which is heartening but i am seeing inconsistent behaviors between two identical servers with identical settings for invocation of the same exact model as well as differenes in execution between restarts of the docker container running it.

[guoqingbao]

The recommend chunk size is 8k while you can disable it by setting 0.

[guoqingbao]

@guoqingbao : q4_1, q4_0, and q5_1 are all susceptible to the large input bug:

$ aichat explain what this diff does: "$(head -1000 large-diff.patch)"
4033874943740755978494570007102433597777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777777Error: Aborted.
it can actually register and error but its not a crash and no stack trace present:

candle-vllm-swe | 2025-08-17 10:36:40 WARN candle_vllm::openai::pipelines::llm_engine New Request with length 11890 tokens (cmpl-192d439b-7fec-49ea-9441-665fc8313b2e).
candle-vllm-swe | 2025-08-17 10:36:40 WARN candle_vllm::openai::pipelines::llm_engine Sending 1 tasks to 3 subprocesses
candle-vllm-swe | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Send stream response error! (sequence id 2)
candle-vllm-swe | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Decoding 279 tokens finished in 7 seconds (cmpl-192d439b-7fec-49ea-9441-665fc8313b2e)
candle-vllm-swe | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine Sending abort message (1 sequence(s)) to subprocesses!
candle-vllm-swe | 2025-08-17 10:37:30 WARN candle_vllm::openai::pipelines::llm_engine engine.write write for aborted_sequences
which seems to be related to setting the prefill chunk size (32k is what i'm trying to set as the lower threshold) - increasing the prefill size appears to make the problem above far more likely to occur. Might be useful to have the option of disabling prefill chunking (fallback to original behavior) by setting it to zero; at least to help narrow down localization of concerns like this.

Some of the gguf formats may not satisfy the accuracy requirements, while, this can be mitigated with mixed precision. Currently, we isq quant all weights to given format but certain weights, like rmsnorm, requires higher precision formats. This can be improved.

[sempervictus]

Interesting thank you. Will do some reading :-)

Any thoughts on native q8_1 for the added precision relative to q8_0? Looks pretty straightforward to implement just wondering why candle-vllm stops at 8_0 (is there something past the linear+model changes which would be impacted?).

[guoqingbao]

Interesting thank you. Will do some reading :-)

Any thoughts on native q8_1 for the added precision relative to q8_0? Looks pretty straightforward to implement just wondering why candle-vllm stops at 8_0 (is there something past the linear+model changes which would be impacted?).

Candle use q8_1 for input quant because the underlying cuda kernels accept q8_1 inputs and other quantized weights. q8_1 can also be used as weight format and we just need some relevant kernels.

[sempervictus]

Prefill chunking definitely helps w/ OOMs on FP16 and tool use works across the board that way - this is test construction remedy on a websearch lib i'm toying with for mistral.rs use and rust-based agents/MCPs:

Looking at the code more carefully, I see the issue. The problem is that the mock server is expecting a JSON response but the test is trying to parse it as a SearXNGResponse struct, and there's likely a mismatch in the expected response structure or the way the mock is 
set up. 

Looking at the error and the implementation, I need to check if the mock server is actually returning what the client expects. Let me first examine the full structure of the response and then fix the test properly.

Let me also look at the actual search method implementation to understand what's happening:
⏺ [09:39:11] Read src/lib.rs [Range 150-200]
Let me look at the actual search method implementation to understand how it makes the request:
⏺ [09:39:34] Read src/lib.rs [Range 100-150]
Now I need to see the complete search method implementation to understand how it makes the request:
⏺ [09:39:57] Read src/lib.rs [Range 180-220]
Let me see the actual request part:
⏺ [09:40:36] Read src/lib.rs [Range 220-240]
⏺ [09:40:43] Read src/lib.rs [Range 240-280]
⏺ [09:40:51] Read src/lib.rs [Range 280-320]
⏺ [09:40:58] Read src/lib.rs [Range 1-40]

we're no longer seeing the <tool info> stuff but actual execution of the tools available.

Requires a minimum of 4 32G GPUs so i can't quite check on the 2-way but it does indicate that we're getting corruption in the quantization process or the fused moe mechanism itself. I am able to reproduce this on a single GPU with --m unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF --f DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf which as i understand it uses the R1 fusedmoe implementation vs the qwen3 one and while its extremely fast, it also fails to use tools correctly but doesn't produce the myriad textual artifacts we see with q3coder.

So far the symptoms i've seen are errant EOS tokens, substitution of > with }, and the hallucinatory repetition thing especially with large volumes of data. Wondering if a warning should be printed for users of quantized q3moe until this is addressed - too many horror stories of agents doing bad things to code and this model is extremely likely to be used in that way.

[guoqingbao]

Prefill chunking definitely helps w/ OOMs on FP16 and tool use works across the board that way - this is test construction remedy on a websearch lib i'm toying with for mistral.rs use and rust-based agents/MCPs:

Looking at the code more carefully, I see the issue. The problem is that the mock server is expecting a JSON response but the test is trying to parse it as a SearXNGResponse struct, and there's likely a mismatch in the expected response structure or the way the mock is
set up.

Looking at the error and the implementation, I need to check if the mock server is actually returning what the client expects. Let me first examine the full structure of the response and then fix the test properly.

Let me also look at the actual search method implementation to understand what's happening:
⏺ [09:39:11] Read src/lib.rs [Range 150-200]
Let me look at the actual search method implementation to understand how it makes the request:
⏺ [09:39:34] Read src/lib.rs [Range 100-150]
Now I need to see the complete search method implementation to understand how it makes the request:
⏺ [09:39:57] Read src/lib.rs [Range 180-220]
Let me see the actual request part:
⏺ [09:40:36] Read src/lib.rs [Range 220-240]
⏺ [09:40:43] Read src/lib.rs [Range 240-280]
⏺ [09:40:51] Read src/lib.rs [Range 280-320]
⏺ [09:40:58] Read src/lib.rs [Range 1-40]
we're no longer seeing the <tool info> stuff but actual execution of the tools available.

Requires a minimum of 4 32G GPUs so i can't quite check on the 2-way but it does indicate that we're getting corruption in the quantization process or the fused moe mechanism itself. I am able to reproduce this on a single GPU with --m unsloth/DeepSeek-R1-0528-Qwen3-8B-GGUF --f DeepSeek-R1-0528-Qwen3-8B-Q4_K_M.gguf which as i understand it uses the R1 fusedmoe implementation vs the qwen3 one and while its extremely fast, it also fails to use tools correctly but doesn't produce the myriad textual artifacts we see with q3coder.

So far the symptoms i've seen are errant EOS tokens, substitution of > with }, and the hallucinatory repetition thing especially with large volumes of data. Wondering if a warning should be printed for users of quantized q3moe until this is addressed - too many horror stories of agents doing bad things to code and this model is extremely likely to be used in that way.

There are some reports that qwen3 coder may not work well with tool usage QwenLM/Qwen3-Coder#499

DeepSeek-R1-0528-Qwen3-8B: this model is not a MoE model, it is a LLaMa dense model.

[guoqingbao]

It might not be a problem with chunked prefill if the fp16 model can work well. I think it's an accuracy issue since V100 does not support bf16 and intermediate computations relying on higher precision values, f16 may works in certain cases but it can lose accuracy when the context getting larger and larger, accumulation of accuracy loss can lead to model corruption. We can make some changes to isq for not quantizing normalization weights, but I'm not sure if that can fully address this issue.