the result about qwen3-8b, ARC-C

[coding-rw]

Hello,

I conducted an evaluation of Qwen3-8B on the ARC-C dataset using two different methods. The results show a significant drop in accuracy with the latent_mas method compared to the baseline. Below are the commands used and the corresponding results:

Baseline Method

python run.py --method baseline --model_name Qwen/Qwen3-8B --task arc_challenge --max_samples -1 --use_vllm --max_new_tokens 2048 --gpu_memory_utilization 0.9

Result:

{
  "method": "baseline",
  "model": "Qwen3-8B",
  "split": "test",
  "seed": 42,
  "max_samples": 1172,
  "accuracy": 0.9198,
  "correct": 1078,
  "total_time_sec": 1971.0141,
  "time_per_sample_sec": 1.6818
}

Latent MAS Method

CUDA_VISIBLE_DEVICES=0,1 python run.py --method latent_mas --model_name Qwen/Qwen3-8B --task arc_challenge --prompt sequential --max_samples -1 --max_new_tokens 2048 \
  --use_vllm \
  --use_second_HF_model \
  --enable_prefix_caching \
  --device2 cuda:1

Result:

{
  "method": "latent_mas",
  "model": "Qwen3-8B",
  "split": "test",
  "seed": 42,
  "max_samples": 1172,
  "accuracy": 0.7108,
  "correct": 833,
  "total_time_sec": 1754.6171,
  "time_per_sample_sec": 1.4971
}

Issue

The baseline achieves an accuracy of ~92.0%, while the latent_mas method drops to ~71.1%. Could you please help me understand if I made any mistakes in the setup or if this is an expected behavior? Any insights would be greatly appreciated.

Thank you!

[coding-rw]

and here is the env

transformers==4.57.1
torch
numpy
tqdm
accelerate
datasets
vllm==0.11.0
matplotlib

[xiyuanyang45]

Hi @coding-rw,

Thanks for your interest in our work! To reproduce the ARC-C results on Qwen3-8B, please run the following command:

python run.py --method latent_mas --model_name Qwen/Qwen3-8B --task arc_challenge --max_samples -1 --prompt sequential --max_new_tokens 2048 --generate_bs 30

Regarding the command you shared, we noticed that you utilized the vLLM backend instead of the HF version. Since our vLLM-based implementation requires modifications to internal packages, we haven’t released that part of the code yet. As a result, directly substituting vLLM for the HF backend at this stage may lead to performance degradation due to internal inconsistencies. We are actively working on this and will release the complete vLLM backend support shortly.

Please feel free to reach out if you have any further questions!

[coding-rw]

Hi @xiyuanyang45 ,

we test baseline and latent_mas again without vLLM, and we get the result as follows:

93.25 > 92.83

{"method": "baseline", "model": "Qwen/Qwen3-8B", "split": "test", "seed": 42, "max_samples": 1172, "accuracy": 0.9325938566552902, "correct": 1093, "total_time_sec": 6580.7602, "time_per_sample_sec": 5.615}

{"method": "latent_mas", "model": "Qwen/Qwen3-8B", "split": "test", "seed": 42, "max_samples": 1172, "accuracy": 0.9283276450511946, "correct": 1088, "total_time_sec": 5839.912, "time_per_sample_sec": 4.9829}

Could you please help me understand if I made any mistakes in the setup or if this is an expected behavior? Any insights would be greatly appreciated.