Skip to content

add testing for Qwen3.5-0.8B to GSM8K eval #36291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
puririshi98 wants to merge 16 commits into
base: main
Choose a base branch
from
+7 −1
Open

add testing for Qwen3.5-0.8B to GSM8K eval #36291

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
877c553
Create test-qwen-deepseek-1.5-accuracy.py
puririshi98 Mar 6, 2026
27705ef
Rename test-qwen-deepseek-1.5-accuracy.py to test-qwen-deepseek-1.5b-…
puririshi98 Mar 9, 2026
46cb85c
Update test-qwen-deepseek-1.5b-accuracy.py
puririshi98 Mar 9, 2026
8071388
Create __init__.py
puririshi98 Mar 9, 2026
043a58f
Update test-qwen-deepseek-1.5b-accuracy.py
puririshi98 Mar 9, 2026
fd677e8
Merge branch 'main' into patch-3
puririshi98 Mar 9, 2026
7f06fb3
Add DeepSeek R1 Distill accuracy test using vLLM's GSM8K harness
puririshi98 Mar 9, 2026
5766643
Merge branch 'main' into patch-3
puririshi98 Mar 9, 2026
cfe9be2
Merge branch 'main' into patch-3
puririshi98 Mar 9, 2026
807825b
Delete tests/v1/attention/test-qwen-deepseek-1.5b-accuracy.py
puririshi98 Mar 9, 2026
445fb44
Delete tests/v1/attention/__init__.py
puririshi98 Mar 9, 2026
89c2828
Update models-small.txt
puririshi98 Mar 9, 2026
b20ef27
Create Qwen3.5-0.8B.yaml
puririshi98 Mar 9, 2026
1a50c2e
Merge branch 'main' into patch-3
puririshi98 Mar 10, 2026
6b8d135
Merge branch 'main' into patch-3
puririshi98 Mar 10, 2026
afbaba2
Merge branch 'main' into patch-3
puririshi98 Mar 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions tests/evals/gsm8k/configs/Qwen3.5-0.8B.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
model_name: "Qwen/Qwen3.5-0.8B"
accuracy_threshold: 0.375
num_questions: 1319
num_fewshot: 5
server_args: "--enforce-eager --max-model-len 4096"
3 changes: 2 additions & 1 deletion tests/evals/gsm8k/configs/models-small.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ Llama-3-8B-Instruct-nonuniform-CT.yaml
Qwen2.5-VL-3B-Instruct-FP8-dynamic.yaml
Qwen1.5-MoE-W4A16-CT.yaml
DeepSeek-V2-Lite-Instruct-FP8.yaml
Qwen3-30B-A3B-MXFP4A16.yaml
Qwen3-30B-A3B-MXFP4A16.yaml
Qwen3.5-0.8B.yaml
Loading