About the reproduction issue.

[lqcStar]

Hello, using your code, the model I trained gets much worse metrics than those reported in the paper. GPT-2 only achieves 37% accuracy on GSM8K (when trained only on GSM8K-AUG and then evaluated). I’d like to ask whether there are any training tricks involved, or whether it’s necessary to train on all datasets and then evaluate on GSM8K to reach the reported 43.7% accuracy.
Additionally, I’d like to ask: since the open-source GSM8K-AUG-NL only provides a training set, how should I perform evaluation?

[zhenyi4]

I am not sure if you are using the correct transformers version. It may affect some accuracy because something may mistmatch in my modeling script. Can you just quickly check if you can get the same test accuarcy when loading my trained checkpoint (https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) by running https://github.com/zhenyi4/codi/blob/main/scripts/test_gpt2.sh.

GSM8K-AUG-NL is derived from GSM8k, which does not contaminate the test set of GSM8k, so we test on GSM8k's test set (https://huggingface.co/datasets/openai/gsm8k) in the paper. You can simply run the test_gpt2.sh mentioned above.