[Bug]: Accuracy Issue with FlashMLA Sparse on DeepSeek V3.2

[wzhao18]

Your current environment

The output of python collect_env.py

Your output of `python collect_env.py` here

🐛 Describe the bug

I ran some eval for Deepseek V3.2 comparing flashMLA and flashinfer on both bf16 and fp8, and find that flashMLA has noticeably worse accuracy than flashinfer, particularly at fp8. Note flashMLA fp8 uses fp8_ds_mla whereas flashinfer uses the standard fp8 format, which could be the reason for its suboptimal performance. However, even for f16, the flashMLA performance is no better than flashinfer across the board. This may be indicative of potential bug in flashMLA integration.

KV Cache	Backend	Benchmark	pass@1 (avg-32)	majority@32	pass@32
bf16	FlashMLA	AIME25	88.33%	93.33%	93.33%
		GPQA-diamond	82.89%	85.86%	96.46%
bf16	FlashInfer	AIME25	90.83%	93.33%	100.00%
		GPQA-diamond	83.14%	87.63%	97.47%
fp8_ds_mla	FlashMLA	AIME25	81.98%	88.33%	90.00%
		GPQA-diamond	78.55%	81.82%	94.95%
fp8	FlashInfer	AIME25	89.17%	93.33%	100.00%
		GPQA-diamond	83.55%	86.36%	96.46%

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[wzhao18]

Details to reproduce the eval:

F16 KV Cache (FlashMLA):

vllm serve nvidia/DeepSeek-V3.2-NVFP4 \
    --trust-remote-code \
    --gpu-memory-utilization 0.8 \
    --stream-interval 20 \
    --kv-cache-dtype bfloat16 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v32 \
    --tool-call-parser deepseek_v32 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v3 \
    --default-chat-template-kwargs '{"thinking": true}'

F16 KV Cache (FlashInfer)

vllm serve nvidia/DeepSeek-V3.2-NVFP4 \
    --trust-remote-code \
    --gpu-memory-utilization 0.8 \
    --stream-interval 20 \
    --kv-cache-dtype bfloat16 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v32 \
    --tool-call-parser deepseek_v32 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v3 \
    --default-chat-template-kwargs '{"thinking": true}' \
    --attention-backend FLASHINFER_MLA_SPARSE

NVFP4 Weight + F8 KV Cache (FlashMLA)

vllm serve nvidia/DeepSeek-V3.2-NVFP4 \
    --trust-remote-code \
    --gpu-memory-utilization 0.8 \
    --stream-interval 20 \
    --kv-cache-dtype fp8 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v32 \
    --tool-call-parser deepseek_v32 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v3 \
    --default-chat-template-kwargs '{"thinking": true}'

NVFP4 Weight + F8 KV Cache (FlashInfer)

vllm serve nvidia/DeepSeek-V3.2-NVFP4 \
    --trust-remote-code \
    --gpu-memory-utilization 0.8 \
    --stream-interval 20 \
    --kv-cache-dtype fp8 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --tokenizer-mode deepseek_v32 \
    --tool-call-parser deepseek_v32 \
    --enable-auto-tool-choice \
    --reasoning-parser deepseek_v3 \
    --default-chat-template-kwargs '{"thinking": true}' \
    --attention-backend FLASHINFER_MLA_SPARSE

Eval (with Nemo-Skills):

MODEL="nvidia/DeepSeek-V3.2-NVFP4"
PORT=8000

echo "Starting AIME25 evaluation with model $MODEL on port $PORT using backend $BACKEND @ AVG-2..."
ns eval \
  --benchmarks=aime25:2 \
  --server_type=vllm \
  --model=$MODEL \
  --server_address=http://localhost:${PORT}/v1 \
  ++max_concurrent_requests=512 \
  ++server.api_key=dummy \
  ++inference.tokens_to_generate=150000 \
  ++inference.temperature=1.0 \
  ++inference.top_p=0.95

echo "Starting GPQA-Diamond evaluation with model $MODEL on port $PORT using backend $BACKEND @ AVG-2..."
ns eval \
  --benchmarks=gpqa:2 \
  --split diamond \
  --server_type=vllm \
  --model=$MODEL \
  --server_address=http://localhost:${PORT}/v1 \
  ++max_concurrent_requests=512 \
  ++server.api_key=dummy \
  ++inference.tokens_to_generate=150000 \
  ++inference.temperature=1.0 \
  ++inference.top_p=0.95

[LopezCastroRoberto]

I think something might be broken with DSv3.2 accuracy. The GSM8K evaluation scores are dropping, even though the accuracy should be around ~0.929.

vllm serve nvidia/DeepSeek-V3.2-NVFP4 -tp 4

python tests/evals/gsm8k/gsm8k_eval.py 

Results:
Accuracy: 0.043
Invalid responses: 0.388
Total latency: 77.347 s
Questions per second: 17.053
Total output tokens: 331136
Output tokens per second: 4281.182

I double-checked that this bug wasn’t introduced by my recent PRs (#35290 and #34917). The issue appears to have been introduced earlier.

[wzhao18]

@LopezCastroRoberto This seems a total accuracy collapse caused by a bug. I think my benchmark was collected ok (without a major bug) given it achieves > 80% on GPQA diamond.

[wzhao18]

@LopezCastroRoberto I think your problem is that you did not set the tokenizer. I tried the following on latest main:

vllm serve nvidia/DeepSeek-V3.2-NVFP4 \ \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --tokenizer-mode deepseek_v32 \
    --tensor-parallel-size 8

And it is working fine:

python tests/evals/gsm8k/gsm8k_eval.py 
Running GSM8K evaluation: 1319 questions, 5-shot
Evaluating: 100%|█████████████████████████████████████████████████████████████████| 1319/1319 [00:42<00:00, 30.72it/s]

Results:
Accuracy: 0.914
Invalid responses: 0.001
Total latency: 42.949 s
Questions per second: 30.711
Total output tokens: 119165
Output tokens per second: 2774.539

[LopezCastroRoberto]

@wzhao18 no, it is not about the tokenizer, the bug seems to be on PR #28044, but there are already bugfixes for this: #36618 #36595 Thanks!