Add Terminal Bench 2.0 (Harbor) support and refactor eval runner #504

XinyuJiangCMU · 2026-01-22T05:16:20Z

This is a follow-up PR to #447, which integrated Terminal Bench (TB) v1.0 into Miles.

📝 Summary

This PR adds support for Terminal Bench 2.0 via Harbor (harbor run) and refactors the eval pipeline around a unified runner abstraction, allowing TB 1.0 and TB 2.0 to be selected via configuration only.

Add support for Terminal Bench 2.0 via Harbor by introducing runner: harbor in the TB eval delegate, while keeping TB 1.0 (runner: tb) fully supported.
Introduce a unified runner abstraction with runner_kwargs to pass CLI flags to both tb run and harbor run without expanding client/server schemas.
Refactor the eval server by extracting runner command construction and metrics parsing logic from tb_server.py into utils/ for better separation of concerns and extensibility.
Preserve backward compatibility: existing TB 1.0 workflows continue to work unchanged, and switching between TB 1.0 and TB 2.0 is configuration-only.

🤝 Collaborators

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com> Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>

gemini-code-assist · 2026-01-22T05:16:42Z

Summary of Changes

Hello @XinyuJiangCMU, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the evaluation capabilities by integrating Terminal Bench 2.0 (Harbor) into the existing framework. It streamlines the evaluation process through a new, unified runner abstraction, allowing users to switch between different Terminal Bench versions simply by adjusting configuration. The changes also improve the maintainability and extensibility of the evaluation server by modularizing key functionalities.

Highlights

Terminal Bench 2.0 (Harbor) Support: Integrated support for Terminal Bench 2.0 via Harbor, allowing evaluation runs using harbor run.
Unified Eval Runner Abstraction: Refactored the evaluation pipeline to use a unified runner abstraction, enabling selection between Terminal Bench 1.0 and 2.0 through configuration.
Flexible CLI Argument Passing: Introduced runner_kwargs to pass arbitrary CLI flags to both tb run and harbor run without modifying client/server schemas.
Improved Modularity: Extracted runner command construction and metrics parsing logic from tb_server.py into dedicated utils/ modules for better separation of concerns and extensibility.
Backward Compatibility: Ensured that existing Terminal Bench 1.0 workflows remain fully supported and functional.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds support for Terminal Bench 2.0 (Harbor) and refactors the evaluation pipeline. The changes introduce a unified runner abstraction, allowing selection between TB 1.0 and TB 2.0 via configuration. The eval server logic is also refactored for better separation of concerns.

My review focuses on the robustness and correctness of the new implementation. I've identified a few areas where the error handling and fallback logic could lead to incorrect or misleading results. Specifically, the handling of missing result files and the selection of evaluation data in the metrics parsing could be made more strict to prevent silent failures. I've also included a minor suggestion to improve a shell script and fix a typo in the documentation.

Overall, this is a solid contribution that significantly extends the evaluation capabilities. Addressing the points below will improve the reliability of the new evaluation pipeline.

gemini-code-assist · 2026-01-22T05:18:50Z

examples/eval/terminal_bench/tb_server.py

+            if not metrics_path.exists():
+                fallback = TerminalBenchEvaluator._find_latest_result(
+                    Path(payload.output_path or "jobs").expanduser()
+                )
+                if fallback is not None:
+                    metrics_path = fallback
+            if not metrics_path.exists():
+                logger.warning("Results file missing at %s", metrics_path)
+                return {}


The fallback logic to find the latest result.json is risky. If a job fails to produce a results file, it could silently pick up the results from a previous or unrelated run, leading to incorrect metrics being reported. This can mask failures and produce misleading evaluation results. It would be safer to treat a missing results file as a hard error for the job.

if not metrics_path.exists(): logger.error("Results file missing at %s", metrics_path) raise RuntimeError(f"Results file not found at {metrics_path}")

gemini-code-assist · 2026-01-22T05:18:50Z

examples/eval/terminal_bench/utils/metrics.py

+    if entry is None:
+        entry = next(iter(evals.values()))
+    if not isinstance(entry, dict):


The logic to select the evaluation entry from the Harbor results file has a fallback that picks the first available entry if the expected one is not found (entry = next(iter(evals.values()))). This can lead to silently reporting metrics from the wrong evaluation if the file contains multiple entries and the expected one is missing. This should be a hard error to ensure metric correctness.

Suggested change

if entry is None:

entry = next(iter(evals.values()))

if not isinstance(entry, dict):

if entry is None:

available_keys = ", ".join(evals.keys())

raise KeyError(

f"Could not find any of the candidate eval keys {candidates} in result file. "

f"Available keys: {available_keys}"

)

if not isinstance(entry, dict):

gemini-code-assist · 2026-01-22T05:18:51Z

examples/eval/scripts/terminal_bench/run-eval-tb-qwen.sh

+sleep 3
+pkill -9 ray
+pkill -9 python


These lines appear to be redundant. The pkill commands on lines 10-11 should have already terminated any lingering ray and python processes after ray stop --force. You can remove these three lines for a cleaner and more concise script.

gemini-code-assist · 2026-01-22T05:18:51Z

examples/eval/terminal_bench/README.md

+eval config (examples use `9051`):
+
+```bash
+python miles/examples/eval/terminal_bench/tb_server.py --host 0.0.0.0 --port 9051 


There is a trailing space in the command example, which could be confusing or cause copy-paste errors for users. It's best to remove it.

Suggested change

python miles/examples/eval/terminal_bench/tb_server.py --host 0.0.0.0 --port 9051

python miles/examples/eval/terminal_bench/tb_server.py --host 0.0.0.0 --port 9051

JessicaJiang-123 and others added 8 commits January 14, 2026 22:20

Integrate Terminal Bench into Miles

6a096a7

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com> Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>

update readme in terminal-bench eval

592fd6a

Merge branch 'main' into main

48398e4

fix formatting

81f85d8

refactor to support harbor runner and tb runner

62f2e0f

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com> Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>

eval: support generic param passthrough via runner_kwargs (#3)

9fbeee2

Co-authored-by: Zhiyao Jiang <jessicajiang324@gmail.com> Co-authored-by: Xinyu Jiang <xinyuj2@andrew.cmu.edu>

update readme

f1275e6

clean

91ce722

gemini-code-assist bot reviewed Jan 22, 2026

View reviewed changes

Merge branch 'main' into support-tb-v2

13e5439

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Terminal Bench 2.0 (Harbor) support and refactor eval runner #504

Add Terminal Bench 2.0 (Harbor) support and refactor eval runner #504

Uh oh!

XinyuJiangCMU commented Jan 22, 2026

Uh oh!

gemini-code-assist bot commented Jan 22, 2026

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot Jan 22, 2026

Uh oh!

gemini-code-assist bot Jan 22, 2026

Uh oh!

gemini-code-assist bot Jan 22, 2026

Uh oh!

gemini-code-assist bot Jan 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	python miles/examples/eval/terminal_bench/tb_server.py --host 0.0.0.0 --port 9051
	python miles/examples/eval/terminal_bench/tb_server.py --host 0.0.0.0 --port 9051

Add Terminal Bench 2.0 (Harbor) support and refactor eval runner #504

Are you sure you want to change the base?

Add Terminal Bench 2.0 (Harbor) support and refactor eval runner #504

Uh oh!

Conversation

XinyuJiangCMU commented Jan 22, 2026

📝 Summary

🤝 Collaborators

Uh oh!

gemini-code-assist bot commented Jan 22, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Jan 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants