feat: add ContextBench cross-validation pipeline for baseline vs MCP comparison

Adds 5 scripts to run Harbor's task-solving agent on ContextBench's
1,136 human-annotated SWE-bench tasks and evaluate retrieval quality:

- select_contextbench_pilot.py: stratified task selection + mirror manifest
- scaffold_contextbench_tasks.py: converts parquet → Harbor task dirs
- convert_harbor_to_contextbench.py: ATIF traces → ContextBench trajectory format
- compare_contextbench_results.py: side-by-side baseline vs MCP report
- contextbench_pilot_2config.sh: paired execution launcher

The converter handles both local tools (Read/Grep/Glob) and MCP tools
(sg_read_file, sg_keyword_search with "file" key extraction and
startLine/endLine span mapping).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: sjarmak

configs/contextbench_pilot_2config.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# ContextBench Cross-Validation Pilot: baseline + MCP (50 tasks)
+#
+# Runs Harbor task-solving agent on ContextBench SWE-bench tasks in both
+# baseline (full local code) and MCP (Sourcegraph) configurations.
+#
+# Prerequisites:
+#   1. Run scripts/select_contextbench_pilot.py to select tasks
+#   2. Run scripts/create_sg_mirrors.py to create mirrors
+#   3. Wait 24-48h for Sourcegraph indexing
+#   4. Run scripts/scaffold_contextbench_tasks.py to create task dirs
+#
+# Usage:
+#   source .env.local && export HARBOR_ENV=daytona && export DAYTONA_OVERRIDE_STORAGE=10240
+#   bash configs/contextbench_pilot_2config.sh
+
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
+
+source "$SCRIPT_DIR/_common.sh"
+load_credentials
+enforce_subscription_mode
+
+# Selection file produced by scaffold_contextbench_tasks.py
+export SELECTION_FILE="$REPO_ROOT/configs/contextbench_run_selection.json"
+export CATEGORY="staging"
+export MODEL="${MODEL:-anthropic/claude-haiku-4-5-20251001}"
+
+if [ ! -f "$SELECTION_FILE" ]; then
+    echo "ERROR: Selection file not found: $SELECTION_FILE"
+    echo "Run: python3 scripts/scaffold_contextbench_tasks.py first"
+    exit 1
+fi
+
+TASK_COUNT=$(python3 -c "import json; print(len(json.load(open('$SELECTION_FILE'))))")
+echo "=== ContextBench Cross-Validation Pilot ==="
+echo "Tasks:    $TASK_COUNT"
+echo "Configs:  baseline-local-direct + mcp-remote-direct"
+echo "Category: $CATEGORY"
+echo "Model:    $MODEL"
+echo "Env:      ${HARBOR_ENV:-local}"
+echo ""
+
+"$SCRIPT_DIR/run_selected_tasks.sh" \
+    --selection-file "$SELECTION_FILE" \
+    --benchmark ccb_contextbench \
+    --full-config mcp-remote-direct \
+    --category "$CATEGORY"

docs/ops/SCRIPT_INDEX.md

@@ -167,9 +167,11 @@ Generated from `scripts/registry.json` by `scripts/generate_script_index.py`.
 - `scripts/backfill_size_metadata.py` [one_off] - Historical one-off script: backfill size metadata.
 - `scripts/backfill_triage_from_manifest.py` [one_off] - Historical one-off script: backfill triage from manifest.
 - `scripts/check_harness_readiness.py` - Utility script for check harness readiness.
+- `scripts/compare_contextbench_results.py` - Utility script for compare contextbench results.
 - `scripts/compute_bootstrap_cis.py` - Utility script for compute bootstrap cis.
 - `scripts/context_retrieval_agent.py` - Utility script for context retrieval agent.
 - `scripts/control_plane.py` - Utility script for control plane.
+- `scripts/convert_harbor_to_contextbench.py` - Utility script for convert harbor to contextbench.
 - `scripts/cross_validate_oracles.py` - Utility script for cross validate oracles.
 - `scripts/daytona_poc_runner.py` - Utility script for daytona poc runner.
 - `scripts/daytona_runner.py` - Utility script for daytona runner.
@@ -208,10 +210,12 @@ Generated from `scripts/registry.json` by `scripts/generate_script_index.py`.
 - `scripts/rerun_zero_mcp_tasks.sh` [one_off] - Historical one-off script: rerun zero mcp tasks.
 - `scripts/rescore_difficulty.py` - Utility script for rescore difficulty.
 - `scripts/run_judge.py` - Utility script for run judge.
+- `scripts/scaffold_contextbench_tasks.py` - Utility script for scaffold contextbench tasks.
 - `scripts/scaffold_feature_tasks.py` - Utility script for scaffold feature tasks.
 - `scripts/scaffold_refactor_tasks.py` - Utility script for scaffold refactor tasks.
 - `scripts/scan_swebench_errors.py` - Utility script for scan swebench errors.
 - `scripts/sdlc_anomaly_scan.py` - Utility script for sdlc anomaly scan.
+- `scripts/select_contextbench_pilot.py` - Utility script for select contextbench pilot.
 - `scripts/smoke_artifact_verifier.py` - Utility script for smoke artifact verifier.
 - `scripts/verify_retrieval_eval_smoke.py` - Utility script for verify retrieval eval smoke.