1st token latency has poor performance than other framework

[Light-Travlling]

when i test the model "DeepSeek-R1-Distill-Qwen-7B", the TTFT metrix worse than openvino，I don't know if it's normal. If so, is there any way to improve this performance

Environment：
CPU：2x8592+
Memory: 16x Hynix HMCG94AGBRA179N 64G DDR5 2Rx4

Benchmark command:

[Duyi-Wang]

Dtype? SETP? -s means how many sockets(cpu nodes) will be used.

[Light-Travlling]

yes，that’s all right, my -s is 2, the dtype is the same as other framework

[Light-Travlling]

the warmup is default，there be any impact？

[Duyi-Wang]

What's your dtype?

[Light-Travlling]

fp16

[Light-Travlling]

1. i test only use one node 0 to test qwen3-8B "numactl --all -C 0-31 -m 0 python /home/tzk/AI_Test/xFasterTransformer/benchmark/benchmark.py --model_name qwen3-4B --token_path /data/Model_File/Qwen3-4B --model_path /data/Model_File/Qwen3-4B-xft --prompt_path /home/tzk/AI_Test/xFasterTransformer/benchmark/prompt.json --batch_size 2 --iteration 1 --dtype bf16 --token_in 32 --token_out 32 --sonnet_prefix_len 200 --sonnet_count 20"

[Duyi-Wang]

Use BF16

[Light-Travlling]

the other framework use fp16 could get the better result, why

[Light-Travlling]

[Duyi-Wang]

Since we focus on BF16 and didn't optimize fp16. What's dtype used in benchmark cmd. The weight_data_type will be convert to dtype in runtime.

[Light-Travlling]

ok, thanks

[Light-Travlling]

we have tested the qwen3-8B model, using fp16, the xft also has poor performance than openvino and vllm+ipex in TTFT