⚡ Group Contrastive Policy Optimization (GCPO)
Official repository of the paper: GCPO: When Contrast Fails, Go Gold
GCPO (Group Contrastive Policy Optimization) is a novel reinforcement learning algorithm designed to enhance the reasoning capabilities of language models, especially in scenarios where the model fails to generate correct responses. Unlike previous methods like GRPO, which rely solely on the model’s own rollouts, GCPO introduces Golden Answers (GAs) — external reference answers — to guide the model’s updates when all sampled responses are incorrect.
This approach ensures:
✅ Full sample utilization — no training data is wasted
🧠 Knowledge transfer — small models learn reasoning strategies from larger models
🚀 Faster convergence and better generalization
- ✅ Golden Answer Injection — handles failure rollouts by injecting correct reference solutions
- ⚖️ Sequence-Level Importance Sampling — stabilizes training under sparse reward settings
- 🔥 Contrastive Optimization — enhances separation between good and bad reasoning traces
- ✨ No KL Penalty Needed — encourages diverse yet effective reasoning behaviors
- 📚 Generalizable — works on math, code, and logical QA tasks
| Item | Status |
|---|---|
| Paper | ✅ Released |
| Model Checkpoints | ✅ Released |
| GCPO Dataset | ⏳ Coming soon |
| Code (Training + Evaluation) | ⏳ Coming soon |
We provide the model weights of GCPO-R1-1.5B, which is trained based on DeepSeek-R1-Distill-Qwen-1.5B using the GCPO algorithm. You can find the model at https://huggingface.co/Ach0/GCPO-R1-1.5B.
To evaluate the model on AIME 2024, run:
python3 vllm_eval.py --model_path Ach0/GCPO-R1-1.5B --test_file dataset/AIME24/aime_2024.jsonl --output_path aime2024_result.jsonl --tensor_parallel_size 4 --mode all
GCPO consistently outperforms DAPO.
If you find this work useful, please cite:
@article{wu2025gcpo,
title={GCPO: When Contrast Fails, Go Gold},
author={Hao Wu and Wei Liu},
journal={arXiv preprint arXiv:2510.07790},
year={2025}
}