Some question about results

[chenzehao82]

According to the reported results, the performance gap between the 4B and 14B models appears to be relatively small on both benchmarks (GSM8K: 82.4 vs. 83.7; HumanEval+: 75.0 vs. 76.8). This seems somewhat inconsistent with the expected scaling benefits of larger models. Could the authors elaborate on the possible reasons why the 14B model does not demonstrate a more substantial improvement over the 4B model? In particular, is this behavior attributable to limitations in training data, optimization or scaling strategy, inference settings (e.g., decoding or prompting), or the evaluation protocol itself?
GSM8K 4B: 82.4, 8B：81.1， 14B：83.7
HumanEval+ 4B: 75.0, 8B：74.4， 14B：76.8

[jiaruzouu]

Hi @chenzehao82 ,

Thanks for your questions about the performance across different sizes of models. I saw that all the results you are referring to in your question are for single models rather than LatentMAS. So I am assuming you are asking why scaling up the backbone model sizes will not result in a larger performance gap.

In our experiments, when evaluating the single model baselines, we compare different-sized models (e.g., Qwen3-4B and Qwen3-14B) under the same prompt setups and report the results. We ensure our implementation follows the common setup and receives similar results as reported in many other papers, such as [1,2].

We agree that, in general, larger models are expected to exhibit stronger performance. On datasets beyond the two you highlighted, we indeed observe clear performance scaling as the model size increases from 4B to 14B. For GSM8K and HumanEval+, the relatively small performance gap may stem from the pre-training data composition and training strategies used for these backbone models. In LatentMAS, we do not introduce any additional parameter updates or fine-tuning of the backbone models. As a result, the behavior you observed here might primarily reflect the characteristics of the off-the-shelf models themselves, rather than effects induced by our method.

References:
[1] Scaling Latent Reasoning via Looped Language Models.
[2] Qwen3 Technical Report.

[chenzehao82]

Dear Jiaru,

I also wanted to ask whether you use think mode when evaluating on AIME.

For reference, with Qwen3 8B, the results differ significantly:

67.3 with think mode

20.9 without think mode

In addition, the paper reports 46.7 for a single Qwen3 8B model. I’m not entirely sure why there is such a large gap between these numbers, so I wanted to clarify the evaluation setting you are using.

Best regards,