Add SDK commit hash to workflow run titles for index benchmarks #351
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds a dynamic run-name to the build images workflows for: - SWE-Bench - SWE-Bench Multimodal - SWT-Bench - Commit0 - GAIA When triggered via workflow_dispatch with an sdk-commit input, the workflow run title will now include the SDK commit hash (e.g., 'Build SWE-Bench Images (SDK: abc1234)'). This makes it easier to identify which SDK version was used for each build at a glance. Fixes #350 Co-authored-by: openhands <openhands@all-hands.dev>
juanmichelini
added a commit
that referenced
this pull request
Jan 23, 2026
The commit0-lite benchmark contains 16 instances total, but only 10 are used as reference (gold) instances for accuracy calculation on the official leaderboard. Issue: PR #351 showed 100.7% accuracy (3652/3628) because we were including all 16 repos instead of just the 10 reference repos, leading to incorrect test totals and impossible >100% accuracy. The 10 reference repos are: - simpy - tinydb - marshmallow - wcwidth - imapclient - voluptuous - jinja - deprecated - cookiecutter - cachetools Changes: - run_infer.py: Filter dataset to only reference repos - eval_infer.py: Skip non-reference repos when calculating totals - Updated total_instances from 16 to 10 - Added comments with links to official leaderboard documentation References: - Leaderboard: https://commit-0.github.io/analysis/ - Breakdown: https://commit-0.github.io/analysis_commit0-lite-plain_fillin/
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a dynamic
run-nameto the build images workflows for index benchmarks, so that the SDK commit hash is displayed in the workflow run title when provided.Changes
Added
run-nameproperty to the following workflow files:build-swebench-images.ymlbuild-swebenchmultimodal-images.ymlbuild-swtbench-images.ymlbuild-commit0-images.ymlbuild-gaia-image.ymlWhen triggered via
workflow_dispatchwith ansdk-commitinput, the workflow run title will now include the SDK commit hash. For example:Build SWE-Bench Images (SDK: abc1234)Build GAIA Images (SDK: main)This makes it easier to identify which SDK version was used for each build at a glance, as mentioned in the issue.
Related Issue
Fixes #350
@juanmichelini can click here to continue refining the PR