Reproducing Results from Table 2 for Llama-1B

[neerajgangwar]

Hi,

Following up on #4, I am having trouble reproducing the results reported in Table 2 of the paper (for Llama-1B). I was able to achieve accuracy comparable to the reported SFT-CoT results after training the model for 25 epochs, so this one is fine.

Could you please confirm the values used for the following arguments for both the Coconut baseline and Coconut with SIM-CoT?

1. max_latent_stage
2. num_epochs

Thanks!
Neeraj

[Wiselnn570]

Hello, thank you for your interest in SIM-CoT.

Regarding the experiments on Llama-1B reported in Table 2, due to

1. the catastrophic forgetting issue when the coconut framework is generalized to larger models (https://openreview.net/forum?id=tG4SgayTtk), and
2. our experimental observation that the training format has a stronger impact on larger models,

we therefore divided the training into three stages.

Experimental Settings

• c_thought: 6
• epochs_per_stage: 1
• max_latent_stage: 1

Training Procedure

1. Stage 1: llm-only Training
   In this stage, all defined implicit tokens participate in training from the beginning, and the explicit reasoning steps are gradually removed. By the end of this stage, all explicit steps are completely removed, and the model performance is approximately ~39.

2. Stage 2: Decoder-only Training
   In this stage, only the decoder is trained. Since llm is frozen, the model performance remains unchanged.

3. Stage 3: Joint Training of Coconut and Decoder
   In this stage, llm and the decoder are trained jointly. After this stage, the model performance reaches approximately ~42.

Additional Recommendation

For larger models (1B, 3B, and 8B), we recommend using SIM-CoT based on CODI, which achieves better performance.

[neerajgangwar]

Thank you for your response. I have follow-up questions based on your response.

You mentioned max_latent_stage to be 1. Does this mean there is only one latent stage, as opposed to 3 stages used in the original Coconut paper or 5 in your paper with GPT-2?

Additionally, Stage 1 under the above-mentioned training procedure is the Coconut baseline training. If it achieves 39%, shouldn't Table 2 mention 39% instead of 33.2% for the Coconut baseline (without SIM-CoT)?

[Wiselnn570]

You mentioned max_latent_stage to be 1. Does this mean there is only one latent stage, as opposed to 3 stages used in the original Coconut paper or 5 in your paper with GPT-2?

Yes.

Additionally, Stage 1 under the above-mentioned training procedure is the Coconut baseline training. If it achieves 39%, shouldn't Table 2 mention 39% instead of 33.2% for the Coconut baseline (without SIM-CoT)?

Our first stage is not Coconut. Under Coconut’s training strategy (with c_thought = 2 and max_latent_stage = 3), the number of implicit tokens is gradually increased via curriculum training, reaching 2 × 3 = 6 total implicit tokens. In contrast, we use six implicit tokens throughout the entire training process from the very beginning.

Moreover, Coconut replaces one explicit reasoning step each time a new group of implicit tokens is appended. Our method does not perform this simultaneous add-and-remove operation. Instead, we only gradually remove explicit reasoning steps, while keeping the number of implicit reasoning tokens fixed.