Oddish

Run evals on Harbor tasks in the cloud.

Oddish extends Harbor with:

• Provider-aware queuing and automatic retries for LLM providers
• Real-time monitoring via dashboard or CLI
• Postgres-backed state and S3 storage for logs

Just replace harbor run with oddish run.

Quick Start

1. Install

uv pip install oddish

2. Generate an API key here

• API key generation is restricted during the beta. To request access, contact the maintainer.

export ODDISH_API_KEY="ok_..."

3. Submit a job

# Run a single agent
oddish run -d terminal-bench@2.0 -a codex -m gpt-5.2-codex --n-trials 3

# Or sweep multiple agents
oddish run -d terminal-bench@2.0 -c sweep.yaml

Example sweep.yaml

agents:
  - name: claude-code
    model_name: anthropic/claude-opus-4-6
    n_trials: 3
  - name: codex
    model_name: openai/gpt-5.4
    n_trials: 3
  - name: terminus-2
    model_name: gemini/gemini-3-flash-preview
    n_trials: 3

4. Monitor Progress

oddish status

Documentation

• core library
• dashboard
• cloud workers
• self-hosting

License

Apache License 2.0