Skip to content

Add AIME2025 benchmark#10

Open
AlexCuadron wants to merge 133 commits intomainfrom
add-aime2025-benchmark
Open

Add AIME2025 benchmark#10
AlexCuadron wants to merge 133 commits intomainfrom
add-aime2025-benchmark

Conversation

@AlexCuadron
Copy link
Owner

@AlexCuadron AlexCuadron commented Mar 5, 2025 •
edited
Loading

This PR adds a new benchmark for AIME2025 based on the existing AIME2024 benchmark. The benchmark evaluates AI agents on problems from the American Invitational Mathematics Examination (AIME) 2025.

Changes

  • Created a new directory structure for AIME2025 benchmark
  • Copied and adapted the necessary files from AIME2024
  • Updated the dataset loading code to use the AIME2025 dataset from Hugging Face
  • Added test scripts for dataset loading and answer extraction
  • Updated all references to AIME2024 in the code
  • Added fallback mechanism to handle the case when Docker is not available
  • Improved robustness of scripts with better error handling

Testing

The benchmark has been tested with:

  • Dataset loading from Hugging Face
  • Answer extraction and normalization
  • Fallback to local runtime when Docker is not available

openhands-agent and others added 30 commits February 25, 2025 04:45
@openhands-agent
feat: Enable llm_completions logging in aider_bench
92e98f6 
- Added update_llm_config_for_completions_logging to imports
- Modified get_config to accept instance parameter
- Updated llm_config to enable completions logging
- Updated process_instance to pass instance to get_config

This change makes aider_bench save llm_completions in the same way as swe_bench,
with completions being saved in {eval_output_dir}/llm_completions/{instance_id}/
@AlexCuadron
Merge pull request #4 from AlexCuadron/feature/aider-bench-llm-comple…
9315619 
…tions-fork

feat: Enable llm_completions logging in aider_bench
@AlexCuadron
Merge remote-tracking branch 'origin/main'
dc59367
@AlexCuadron
Add polyglot benchmark implementation
bc8f20d
@AlexCuadron
Fix argument parser in polyglot benchmark
37ba696
@AlexCuadron
Improve polyglot benchmark path handling and fix logging error
890377d
@AlexCuadron
Add Docker configuration options and troubleshooting guide
8af6f11
@AlexCuadron
Add local Docker image build support for polyglot benchmark
32335ff
@AlexCuadron
Set Docker image to build automatically by default
5610010
@AlexCuadron
Fix Docker build issues by adding unzip and simplifying Gradle instal…
c9e232e 
…lation
@AlexCuadron
Restrict polyglot benchmark to use only the same tools as SWE-Bench (…
97e7ca7 
…execute_bash, finish, str_replace_editor)
@AlexCuadron
Fix runtime completion to use Docker runtime for running tests
44bcb39
@AlexCuadron
Add script to test one instance per language in polyglot benchmark
601da45
@AlexCuadron
Add one-per-language testing mode to polyglot benchmark run_infer.sh
84293fd
@AlexCuadron
Update README with one-per-language testing instructions and command-…
87d9e15 
…line options
@AlexCuadron
Enable LLM completions logging in aider_bench run_infer.py
8a5dc59
@AlexCuadron
Include tools information in evaluation output directory names
8ffe33e
@AlexCuadron
Add evaluation parameter to run_infer.sh scripts for aider_bench and …
d45b98d 
…polyglot_benchmark
@AlexCuadron
Update README files with documentation for the new evaluation parameter
62d2632
@AlexCuadron
Fix output directory detection in evaluation scripts
c8dab2c
@AlexCuadron
Fix LLM completions logging to ensure it's enabled in all benchmarks
fa9a0f8
@AlexCuadron
Improve output directory detection in evaluation scripts with better …
8a4ca1e 
…path matching and debugging output
@AlexCuadron
Fix handling of 'eval' parameter to prevent it from being treated as …
a2d7e63 
…an instance ID
@AlexCuadron
Merge pull request #6 from AlexCuadron/polyglot-benchmark-clean
013ff2d 
Add polyglot benchmark implementation
@openhands-agent @AlexCuadron
feat: Enable llm_completions logging in aider_bench
ee6026b 
- Added update_llm_config_for_completions_logging to imports
- Modified get_config to accept instance parameter
- Updated llm_config to enable completions logging
- Updated process_instance to pass instance to get_config

This change makes aider_bench save llm_completions in the same way as swe_bench,
with completions being saved in {eval_output_dir}/llm_completions/{instance_id}/
@AlexCuadron
Add polyglot benchmark implementation
96f6c8a
@AlexCuadron
Fix argument parser in polyglot benchmark
ccff971
@AlexCuadron
Improve polyglot benchmark path handling and fix logging error
e63c293
@AlexCuadron
Add Docker configuration options and troubleshooting guide
3e98953
@AlexCuadron
Add local Docker image build support for polyglot benchmark
95e212b
AlexCuadron added 29 commits March 3, 2025 02:31
@AlexCuadron
Apply temperature settings and boxed answer directive to Math500 benc…
b237ddb 
…hmark
@AlexCuadron
Fix answer normalization to handle currency values properly in Math50…
7be62fc 
…0 and AIME2024 benchmarks
@AlexCuadron
sth
164fcba
@AlexCuadron
Fix overthinking analysis in AIME2024 benchmark
9ad1892
@AlexCuadron
Fix LLM completion method in overthinking analysis
c96c1e1
@AlexCuadron
Implement retry mechanism for overthinking solutions
4b52092
@AlexCuadron
Fix issue with execute_ipython_cell being interpreted as finish funct…
eb68e2a 
…ion call
@AlexCuadron
Add AIME2025 benchmark
55e510d
@AlexCuadron
Fix AIME2025 benchmark to handle Docker availability and improve robu…
450bb27 
…stness
@AlexCuadron
Add prefix-based LLM implementation for AIME2025 benchmark
739c3c0
@AlexCuadron
Update prefix implementation to use <tool> instead of <function>
3ce309a
@AlexCuadron
Update parameter format from <parameter=name>...</parameter> to <name…
cb6decc 
…>...</name>
@AlexCuadron
Fix regex pattern to exclude tool tag when matching parameters
57a450f
@AlexCuadron
Update function names and variables to use 'tool' instead of 'function'
8237b2a
@AlexCuadron
Update import statements in llm.py to use new function names with ali…
af37213 
…ases
@AlexCuadron
Add fallback for older versions of LiteLLM without register_provider
8346553
@AlexCuadron
Add simplified prefix setup and script for AIME2025 benchmark
147364f
@AlexCuadron
Add direct prefix patch for AIME2025 benchmark
a857a62
@AlexCuadron
Resolve merge conflicts in fn_call_converter.py, keeping the tool= fo…
ca5b817 
…rmat
@AlexCuadron
Resolve merge conflicts from openhands-workspace-idr6hq92
831dec1
@AlexCuadron
Update fn_call_converter.py to use <function=...> format instead of <…
ca069ef 
…tool=...> format
@AlexCuadron
Update to use <tool=...> format with <code>...</code> tags
dd10edf
@AlexCuadron
Fix FN_PARAM_REGEX_PATTERN to exclude tool tags
39b4515
@AlexCuadron
Add custom providers and configuration files for AIME2025 benchmark
1cf2075
@AlexCuadron
Add function aliases for backward compatibility
e8e6c3f
@AlexCuadron
Simplify conditional prefix LLM to avoid dependency issues
453c953
@AlexCuadron
Fix custom_qwen_provider to handle missing register_provider attribute
943a268
@AlexCuadron
Add main function to run_infer.py for compatibility with run_with_qwe…
3df8936 
…n.py
@AlexCuadron
Fix run_with_qwen.py to pass through all arguments to run_infer.main()
fa7c282
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@AlexCuadron @openhands-agent @raymyers