Skip to content

README.md

Latest commit

 

History

History
96 lines (70 loc) · 3.54 KB

README.md

File metadata and controls

96 lines (70 loc) · 3.54 KB

Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?

Xiangyang Li · Xiaopeng Li · Kuicai Dong · Quanhu Zhang
Rongju Ruan · Xinyi Dai · Yasheng Wang · Ruiming Tang

📖Paper | 🏠Homepage&Leaderboard | 🤗Huggingface | 👉Github

Code generation is a core capability of large language models (LLMs), yet mainstream benchmarks (e.g., APPs and LiveCodeBench) contain questions with medium-level difficulty and pose no challenge to advanced LLMs. To better reflect advanced reasoning and code generation ability, we introduce Humanity's Last Code Exam (HLCE), comprising 235 of the most challenging problems from the International Collegiate Programming Contest (ICPC World Finals) and the International Olympiad in Informatics (IOI) spanning 2010-2024.

Logo

Why Do We Need HLCE?

With the increasing capabilities of LLMs, many benchmarks have become too easy!

HLCE vs other benchmarks

🛠️Dataset Usage

🔮Dataset Evaluation

Prerequisites

  • Python 3.8 or higher
  • Git

Setup Instructions

  1. Clone the repository:

    git clone git@github.com:Humanity-s-Last-Code-Exam/HLCE.git
cd HLCE

  2. Install the package and its dependencies:

    pip install -e .

  • For IOI, kindly follow these instructions to obtain the definitive evaluation results.

  • For ICPC-World-Finals,kindly follow these instructions to obtain the definitive evaluation results.

📊 Leaderboard

  • If you wish to submit your model to the leaderboard, please follow the instructions.

💾Citation

@misc{li2025humanityscodeexamadvanced,
      title={Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?}, 
      author={Xiangyang Li and Xiaopeng Li and Kuicai Dong and Quanhu Zhang and Rongju Ruan and Xinyi Dai and Xiaoshuang Liu and Shengchun Xu and Yasheng Wang and Ruiming Tang},
      year={2025},
      eprint={2506.12713},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2506.12713}, 
}

📄 License

Code License Data License

Usage and License Notices: The data and code are intended and licensed for research use only.