Modularize Ralph into Python Package

[rjernst]

---

branch: modularize-ralph

Spec: Modularize Ralph into Python Package

Overview

Ralph has grown to ~3000 lines in a single file (scripts/ralph) with ~5000 lines of tests. This spec breaks it into a proper Python package under tools/ralph/ with shared code in tools/libs/dotlib/. The scripts/ralph entry point becomes a thin shell wrapper.

The Git class is duplicated identically in both scripts/ralph and scripts/ta-wt today — extracting it into tools/libs/dotlib/git.py eliminates this duplication and establishes the shared library pattern for the planned ta migration later.

Architecture

tools/
  libs/
    dotlib/                    # shared Python package
      __init__.py              # re-exports: from dotlib.git import Git
      git.py                   # Git class (currently duplicated in ralph + ta-wt)
    pyproject.toml             # name = "dotlib"
  ralph/
    src/
      ralph/                   # ralph Python package
        __init__.py
        __main__.py            # python3 -m ralph entry point
        cli.py                 # arg parsing + main() dispatch
        util.py                # parse_duration, parse_frontmatter, parse_issue_branch
        github.py              # GitHub class with retry logic
        proxy.py               # proxy lifecycle, health, keepalive
        token.py               # Keychain I/O, store/check/get/ensure
        orchestration.py       # resolve_repo, dependencies, worktree, fast-forward
        loop.py                # process_issue + poll_loop
        selftest.py            # selftest orchestration + backend checks
        sandbox/
          __init__.py          # SandboxBackend base class + factory
          docker.py            # DockerSandbox
          tart.py              # TartSandbox
    tests/
      conftest.py              # ralph test fixtures
      test_util.py
      test_github.py
      test_proxy.py
      test_token.py
      test_orchestration.py
      test_loop.py
      test_selftest.py
      test_sandbox_docker.py
      test_sandbox_tart.py
      test_cli.py
      test_integration.py
    pyproject.toml             # name = "ralph", pythonpath = ["src", "../../libs"]
scripts/
  ralph                        # thin shell wrapper (3 lines)

Module dependency graph (arrows = imports from):

cli ──→ token, selftest, loop, proxy, orchestration, util
loop ──→ sandbox, proxy, orchestration, github, util
selftest ──→ sandbox, proxy, token
orchestration ──→ dotlib.git, github, util
sandbox.docker ──→ sandbox base, dotlib.git
sandbox.tart ──→ sandbox base, dotlib.git
sandbox base ──→ token, proxy
proxy ──→ (stdlib only)
token ──→ (stdlib only)
github ──→ (stdlib only)
util ──→ (stdlib only)

Constraint: All Python code remains stdlib-only. No third-party runtime dependencies.

---

1. Directory and Package Structure

• Create tools/libs/dotlib/ and tools/ralph/src/ralph/ and tools/ralph/src/ralph/sandbox/ directories with __init__.py files.
• tools/libs/pyproject.toml: minimal config with name = "dotlib".
• tools/ralph/pyproject.toml: pytest config with pythonpath = ["src", "../../libs"] and testpaths = ["tests"].
• tools/ralph/src/ralph/__main__.py: from ralph.cli import main; main().
• All __init__.py files can be empty initially (populated as modules are extracted).

2. Shell Wrapper

scripts/ralph becomes:

#!/bin/sh
DOTFILES_DIR="$(cd "$(dirname "$0")/.." && pwd)"
PYTHONPATH="${DOTFILES_DIR}/tools/ralph/src:${DOTFILES_DIR}/tools/libs" exec python3 -m ralph "$@"

This replaces the current ~3000-line Python script. The wrapper must resolve correctly both when called directly and via the ~/bin/ralph symlink.

3. Extraction Order and Module Boundaries

Extract in dependency order (leaves first). At each step, move the code and its tests together. After each extraction, all tests must pass.

Leaf modules (no internal dependencies):

• dotlib.git — Git class from lines 35–56. Also update scripts/ta-wt to import from dotlib.git (add sys.path insert at top of ta-wt, remove its local Git class).
• ralph.util — parse_duration, parse_frontmatter, parse_issue_branch.
• ralph.github — GitHub class and _SelftestAbort exception.
• ralph.proxy — constants (DEFAULT_PROXY_PORTS, DEFAULT_PROXY_PORT, MODEL_ALIASES), proxy_script_path, compute_proxy_version, proxy_pid_file, proxy_log_file, proxy_port_for_agent, proxy_health_check, start_proxy, stop_proxy, start_proxy_keepalive, ensure_proxy.
• ralph.token — constants (MS_PER_DAY, DEFAULT_EXPIRY_DAYS), keychain_service_name, read_token_from_keychain, write_token_to_keychain, format_expiry_date, run_claude_setup_token, _parse_and_store_token, store_token, check_token, get_token, ensure_token.

Mid-level modules:

• ralph.sandbox — load_sandbox_config, create_sandbox_backend factory, SandboxBackend base class (with ITERATION_PROMPT).
• ralph.sandbox.docker — DockerSandbox class.
• ralph.sandbox.tart — TartSandbox class.
• ralph.orchestration — resolve_repo, check_dependencies, unblock_ready_specs, ensure_worktree, try_fast_forward, check_dependencies_prereq.

Top-level modules:

• ralph.selftest — selftest, _selftest_docker, _selftest_tart.
• ralph.loop — process_issue, poll_loop.
• ralph.cli — USAGE string, usage(), main().

4. Test Migration

Move tests from tests/test_ralph.py, tests/test_ralph_proxy.py, tests/test_ralph_integration.py into tools/ralph/tests/, splitting by module. Tests import directly from the package (e.g., from ralph.util import parse_duration) — no import_script() hack needed.

Keep tests/test_ralph.bats in place (it tests the CLI entry point end-to-end via the shell wrapper).

Update tests/test_ralph.bats to work with the new shell wrapper entry point.

If tests/conftest.py is still needed by other test files (e.g., test_ta_wt.py), keep it. Remove the ralph-specific parts.

5. ta-wt Integration

Update scripts/ta-wt to import Git from dotlib instead of defining its own copy:

import sys, os
sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "tools", "libs"))
from dotlib.git import Git

Remove the local Git class definition from ta-wt. Run ta-wt tests to verify.

---

Implementation Plan

Step 1: Create directory structure and pyproject.toml files [done]

Files:

• tools/libs/dotlib/__init__.py — empty
• tools/libs/pyproject.toml — minimal project config
• tools/ralph/src/ralph/__init__.py — empty
• tools/ralph/src/ralph/__main__.py — from ralph.cli import main; main()
• tools/ralph/src/ralph/sandbox/__init__.py — empty
• tools/ralph/tests/conftest.py — empty (or minimal fixtures)
• tools/ralph/pyproject.toml — pytest config with pythonpath

Implement:

1. Create all directories and files listed above.
2. tools/libs/pyproject.toml should have [project] name = "dotlib" and version = "0.0.0".
3. tools/ralph/pyproject.toml should have [project] name = "ralph", version = "0.0.0", and [tool.pytest.ini_options] pythonpath = ["src", "../libs"] and testpaths = ["tests"].
   • Note: Changed ../../libs to ../libs — from tools/ralph/, the relative path to tools/libs is ../libs (one level up), not ../../libs (two levels up).

Test:

• cd tools/ralph && python3 -c "import ralph" succeeds (empty package imports).

Verify: Run the import check above. Fix any failures.

Review: Verify all directories and files exist with correct content.

Address feedback: Fix all review findings.

Step 2: Extract dotlib.git and update ta-wt [done]

Files:

• tools/libs/dotlib/git.py — Git class extracted from ralph
• tools/libs/dotlib/__init__.py — re-export Git
• scripts/ta-wt — remove local Git class, import from dotlib
• scripts/ralph — (still the monolith at this step) import Git from dotlib instead of defining locally

Implement:

1. Copy the Git class (lines 35–56 of current ralph) into tools/libs/dotlib/git.py.
2. Update tools/libs/dotlib/__init__.py to from dotlib.git import Git.
3. In scripts/ralph, add sys.path.insert(0, ...) for tools/libs near the top, remove the local Git class, and add from dotlib.git import Git.
4. In scripts/ta-wt, add sys.path.insert(0, ...) for tools/libs near the top, remove the local Git class, and add from dotlib.git import Git.

Test:

• ralph --help still works.
• ta-wt --help still works.
• Existing tests for both ralph and ta-wt pass.

Verify: Run pytest tests/test_ralph.py tests/test_ralph_proxy.py -v and pytest tests/test_ta_wt.py -v. Fix any failures and re-run until all pass.

Review: Verify the Git class was moved cleanly with no behavior changes. Verify both scripts resolve the dotlib path correctly.

Address feedback: Fix all review findings. Re-run tests.

Step 3: Extract leaf modules (util, github, proxy, token) [done]

Files:

• tools/ralph/src/ralph/util.py — pure utility functions
• tools/ralph/src/ralph/github.py — GitHub class + _SelftestAbort
• tools/ralph/src/ralph/proxy.py — proxy lifecycle functions + constants
• tools/ralph/src/ralph/token.py — token management functions + constants
• tools/ralph/tests/test_util.py — tests from test_ralph.py for util functions
• tools/ralph/tests/test_github.py — tests for GitHub class
• tools/ralph/tests/test_proxy.py — tests from test_ralph_proxy.py + test_ralph.py proxy tests
• tools/ralph/tests/test_token.py — tests for token functions

Implement:

1. Extract parse_duration, parse_frontmatter, parse_issue_branch into ralph/util.py.
2. Extract GitHub class and _SelftestAbort into ralph/github.py.
3. Extract all proxy functions and constants (DEFAULT_PROXY_PORTS, DEFAULT_PROXY_PORT, MODEL_ALIASES, proxy_script_path, compute_proxy_version, proxy_pid_file, proxy_log_file, proxy_port_for_agent, proxy_health_check, start_proxy, stop_proxy, start_proxy_keepalive, ensure_proxy) into ralph/proxy.py.
4. Extract all token functions and constants (MS_PER_DAY, DEFAULT_EXPIRY_DAYS, keychain_service_name, read_token_from_keychain, write_token_to_keychain, format_expiry_date, run_claude_setup_token, _parse_and_store_token, store_token, check_token, get_token, ensure_token) into ralph/token.py.
5. In the monolith scripts/ralph, replace the moved code with imports from the new modules.
6. Move corresponding test classes from tests/test_ralph.py and tests/test_ralph_proxy.py into the new test files under tools/ralph/tests/. Update imports to use the package directly (e.g., from ralph.util import parse_duration).

Test:

• All moved tests pass from tools/ralph/tests/.
• Remaining tests in tests/test_ralph.py still pass (they still import the monolith which re-imports from modules).

Verify: Run cd tools/ralph && pytest tests/ -v and pytest tests/test_ralph.py -v. Fix any failures and re-run until all pass.

Review: Verify no functions were lost or duplicated. Verify imports are clean.

Address feedback: Fix all review findings. Re-run tests.

Notes:

• ralph.proxy imports read_token_from_keychain from ralph.token (the spec's dependency graph listed proxy as stdlib-only, but start_proxy requires it).
• The monolith sets __path__ to make sub-module imports work when loaded as the ralph module by test infrastructure.
• tests/test_ralph_proxy.py was NOT moved — it tests docker/agent-loop/proxy/proxy.py, not the ralph proxy module.
• Remaining monolith tests that call extracted functions through imported bindings needed patch path updates: TestMainTokenSubcommands patches → ralph.token.*, TestGitHubRetry patches → ralph.github.*, TestEnsureProxyStaleCleanup patches → ralph.proxy.*.

Step 4: Extract sandbox modules [done]

Files:

• tools/ralph/src/ralph/sandbox/__init__.py — load_sandbox_config, create_sandbox_backend, SandboxBackend base class
• tools/ralph/src/ralph/sandbox/docker.py — DockerSandbox
• tools/ralph/src/ralph/sandbox/tart.py — TartSandbox
• tools/ralph/tests/test_sandbox_docker.py — Docker sandbox tests
• tools/ralph/tests/test_sandbox_tart.py — Tart sandbox tests

Implement:

1. Move SandboxBackend base class (with ITERATION_PROMPT and sandbox_name), load_sandbox_config, and create_sandbox_backend factory into ralph/sandbox/__init__.py.
2. Move DockerSandbox into ralph/sandbox/docker.py. Update its imports to use from dotlib.git import Git, from ralph.proxy import ..., from ralph.token import ..., from ralph.sandbox import SandboxBackend.
3. Move TartSandbox into ralph/sandbox/tart.py with similar import updates.
4. Update the factory in sandbox/__init__.py to import from .docker and .tart.
5. In the monolith, replace sandbox code with imports.
6. Move sandbox-related tests into the new test files.

Test:

• All sandbox tests pass from tools/ralph/tests/.
• Remaining monolith tests still pass.

Verify: Run cd tools/ralph && pytest tests/ -v and pytest tests/test_ralph.py -v. Fix any failures and re-run until all pass.

Review: Verify the sandbox hierarchy is intact, factory works, and all backend methods are preserved.

Address feedback: Fix all review findings. Re-run tests.

Notes:

• sandbox/__init__.py imports proxy_health_check from ralph.proxy and read_token_from_keychain from ralph.token for the preflight_check method.
• create_sandbox_backend uses lazy imports to avoid circular imports (from ralph.sandbox.docker import DockerSandbox inside the function body).
• The monolith imports the sandbox classes into its own namespace (from ralph.sandbox.docker import DockerSandbox), so remaining monolith tests that patch ralph.DockerSandbox.* or ralph.TartSandbox.* continue to work.
• Test patch paths were updated: ralph.sandbox.docker.subprocess.run for Docker backend subprocess calls, ralph.sandbox.tart.subprocess.run for Tart backend calls, ralph.sandbox.proxy_health_check and ralph.sandbox.read_token_from_keychain for preflight checks.
• TestTartCheckPrerequisites was moved from the monolith to test_sandbox_tart.py with patch path ralph.sandbox.tart.shutil.which.
• 282 module tests (including 142 Docker + 70 Tart sandbox tests) and 83 monolith tests all pass.

Step 5: Extract orchestration, loop, selftest, and CLI [done]

Files:

• tools/ralph/src/ralph/orchestration.py — repo resolution, dependencies, worktree, fast-forward
• tools/ralph/src/ralph/loop.py — process_issue, poll_loop
• tools/ralph/src/ralph/selftest.py — selftest orchestration
• tools/ralph/src/ralph/cli.py — USAGE, usage(), main()
• tools/ralph/tests/test_orchestration.py
• tools/ralph/tests/test_loop.py
• tools/ralph/tests/test_selftest.py
• tools/ralph/tests/test_cli.py

Implement:

1. Extract resolve_repo, check_dependencies, unblock_ready_specs, ensure_worktree, try_fast_forward, check_dependencies_prereq into ralph/orchestration.py.
2. Extract process_issue, poll_loop into ralph/loop.py.
3. Extract selftest, _selftest_docker, _selftest_tart into ralph/selftest.py.
4. Extract USAGE, usage(), main() into ralph/cli.py.
5. Move all remaining tests from tests/test_ralph.py into the appropriate new test files.
6. Move tests/test_ralph_integration.py to tools/ralph/tests/test_integration.py, updating imports.
7. tests/test_ralph.py should now be empty or nearly so — delete it.
8. Delete tests/test_ralph_proxy.py if all its tests have been moved.

Test:

• All tests pass from tools/ralph/tests/.
• tests/test_ralph.bats still passes.

Verify: Run cd tools/ralph && pytest tests/ -v and bats tests/test_ralph.bats. Fix any failures and re-run until all pass.

Review: Verify the monolith scripts/ralph is now empty of Python code. Verify no test was lost.

Address feedback: Fix all review findings. Re-run tests.

Notes:

• The monolith scripts/ralph is now a thin re-export wrapper: it imports all public names from the extracted modules into its own namespace, so any code that patches ralph.<name> still works.
• tests/test_ralph.py was deleted after all tests were moved to their respective module test files.
• tests/test_ralph_integration.py was moved to tools/ralph/tests/test_integration.py with updated imports (uses package imports instead of import_script()).
• tests/test_ralph_proxy.py was NOT deleted — it tests the Docker proxy server (docker/agent-loop/proxy/proxy.py), not the ralph proxy module.
• Remaining tests from the monolith that belonged to already-extracted modules (TestEnsureProxyStaleCleanup, TestGitHubRetry, TestIterationPrompt) were appended to the appropriate existing test files (test_proxy.py, test_github.py, test_sandbox_docker.py).
• cli.py resolves dotfiles_dir relative to its own file path (../../../.. from tools/ralph/src/ralph/), which matches the repo root.
• Patch paths in tests were updated from ralph.<name> to ralph.<module>.<name> (e.g., ralph.loop.resolve_repo, ralph.selftest.subprocess.run, ralph.cli.sys.argv).
• CLI tests for selftest routing were split from the TestSelftest class into a separate TestMainSelftestRouting class in test_cli.py.
• 365 module tests pass, 6 integration tests skipped (require Docker + token).
• bats test 5 ("ralph fails when gh is not installed") fails in this environment because gh is installed at /usr/bin/gh — this is a pre-existing environment-specific issue, confirmed to fail identically on the pre-extraction monolith.

Step 6: Convert scripts/ralph to shell wrapper [done]

Files:

• scripts/ralph — replace Python monolith with 3-line shell wrapper

Implement:

1. Replace scripts/ralph with the shell wrapper:

   #!/bin/sh
   DOTFILES_DIR="$(cd "$(dirname "$0")/.." && pwd)"
   PYTHONPATH="${DOTFILES_DIR}/tools/ralph/src:${DOTFILES_DIR}/tools/libs" exec python3 -m ralph "$@"

2. Verify tools/ralph/src/ralph/__main__.py exists and contains from ralph.cli import main; main().
3. Update tests/test_ralph.bats if needed to work with the shell wrapper (the CLI behavior should be identical).

Test:

• ralph --help works.
• ralph selftest works.
• tests/test_ralph.bats passes.

Verify: Run ralph --help, bats tests/test_ralph.bats, and cd tools/ralph && pytest tests/ -v. Fix any failures and re-run until all pass.

Review: Verify the shell wrapper resolves paths correctly from both direct invocation and ~/bin symlink.

Address feedback: Fix all review findings. Re-run tests.

Notes:

• The shell wrapper is 10 lines (not 3 as spec suggested) because portable symlink resolution requires a loop — macOS lacks readlink -f.
• The symlink resolution loop follows one-level symlinks using POSIX-compatible readlink + dirname/cd/pwd.
• The __main__.py entry point was already correct from Step 1 (from ralph.cli import main; main()).
• tests/test_ralph.bats required no changes — all 13 non-environment-specific tests pass. Test 5 ("ralph fails when gh is not installed") continues to fail in sandbox environments where gh is installed at /usr/bin/gh (pre-existing, documented in Step 5 notes).
• 365 module pytest tests pass, 6 integration tests skipped.

Step 7: Run all checks [done]

Implement:

1. Run the full test suite: cd tools/ralph && pytest tests/ -v
2. Run bats tests: bats tests/test_ralph.bats
3. Run ta-wt tests: bats tests/test_ta_wt.bats
4. Run shellcheck on the new shell wrapper: shellcheck scripts/ralph
5. Verify ralph --help works.
6. Verify no stale test files remain in tests/ for code that was moved.
7. Fix any failures and commit the fixes.

Verify: All checks pass clean.

Notes:

• Spec said pytest tests/test_ta_wt.py but ta-wt tests are bats tests (tests/test_ta_wt.bats), not pytest.
• ta-wt bats test 35 ("wt remove calls workspace kill") was failing because the test copies ta-wt to a mock directory, breaking the sys.path.insert that resolves dotlib relative to the script's location. Fixed by setting PYTHONPATH in the test's run command so the copied script can find dotlib.
• Ralph bats test 5 ("ralph fails when gh is not installed") continues to fail in sandbox environments where gh is installed — pre-existing, documented in Step 5 notes.
• tests/test_ralph_proxy.py remains in tests/ — it tests the Docker proxy server (docker/agent-loop/proxy/proxy.py), not the ralph proxy module.
• Results: 365 module pytest passed (6 skipped), 13/14 ralph bats passed, 63/66 ta-wt bats passed (3 skipped: no zsh), shellcheck clean.

---

Conventions

• Language: Python 3 (stdlib only, no third-party runtime dependencies)
• Tests: pytest, run via cd tools/ralph && pytest tests/ -v
• Entry point: scripts/ralph is a shell wrapper; actual code lives in tools/ralph/src/ralph/
• Shared code: tools/libs/dotlib/ — only put code here when 2+ tools use it today
• Imports: Use standard package imports (from ralph.util import parse_duration), no import_script() hack
• Error messages: Prefix with ralph: for user-facing errors
• Exit codes: 0=success, 1=runtime error, 2=usage error