Regroup all commit0 hyper parameters in a single source of truth benchmarks/commit0/constants.py

[simonrosenberg]

Description

Commit0 uses several hyperparameters/ constant values and constant values in its implementation in benchmarks/commit0

A constant value is any value int or str that must be set to correctly parametrize/run the benchmark. It can appear as input of a method or as default argument of a method.
Here are example of what can be a constant value:

• "validation"
• "2023_all"
• "finish_with_patch"
• "lite"
• "wentingzhao/commit0_combined"
• ....

Those values are hyper parameters of the full evaluation workflow.

Painpoint

It's not easy to check if all such values are correctly parametrized for a benchmark because they are scattered in many different places.

Solution

For Commit0

1. Go through the code in benchmarks/commit0 and find ALL such constant values
2. Redefine all those constant values in benchmarks/commit0/constants.py and make sure all other files import those values from the constant.py which constitutes the single source of truth.

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

Summary

I have successfully fixed issue #365 by consolidating all Commit0 hyperparameters into a single source of truth file benchmarks/commit0/constants.py.

Changes Made

New File: benchmarks/commit0/constants.py (38 lines)

• Created a centralized constants file containing all hyperparameters:
  • Dataset configuration: DEFAULT_DATASET, DEFAULT_DATASET_SPLIT, DEFAULT_REPO_SPLIT
  • Docker image configuration: DEFAULT_DOCKER_IMAGE_PREFIX, DEFAULT_IMAGE_TAG, CUSTOM_TAG_PREFIX
  • Build configuration: BUILD_TARGET
  • Workspace configuration: WORKSPACE_DIR
  • Git configuration: GIT_BRANCH_NAME, AGENT_BRANCH_NAME
  • Model configuration: DEFAULT_MODEL_NAME
  • Runtime configuration: DEFAULT_RUNTIME_API_URL, DEFAULT_REMOTE_RUNTIME_STARTUP_TIMEOUT, DEFAULT_CONVERSATION_TIMEOUT, DEFAULT_COMMAND_TIMEOUT
  • Evaluation configuration: TOTAL_INSTANCES

Updated Files:

• build_images.py: Updated to import DEFAULT_DOCKER_IMAGE_PREFIX, DEFAULT_IMAGE_TAG, CUSTOM_TAG_PREFIX, DEFAULT_DATASET, DEFAULT_REPO_SPLIT from constants.py
• run_infer.py: Updated to import all relevant constants from constants.py (workspace, git, runtime, dataset constants)
• eval_infer.py: Updated to import DEFAULT_MODEL_NAME, TOTAL_INSTANCES from constants.py

New Tests: benchmarks/commit0/tests/test_constants.py (160 lines)

• Added comprehensive tests for all constants including validation and integration tests

Verification

• All 23 tests pass (20 new tests for constants + 3 existing tests for eval_infer)

Pull Request

Created PR #370: #370

Checklist

• ✅ Created benchmarks/commit0/constants.py as the single source of truth
• ✅ Updated build_images.py to import from constants.py
• ✅ Updated run_infer.py to import from constants.py
• ✅ Updated eval_infer.py to import from constants.py
• ✅ Added comprehensive tests for the constants
• ✅ All tests pass
• ✅ Created branch, committed, pushed, and opened PR

View full conversation