From 1df737a60c344a3d0898b868e03e2912da6c720b Mon Sep 17 00:00:00 2001 From: Adrian Sutton Date: Thu, 2 Apr 2026 01:35:15 +0000 Subject: [PATCH 1/2] docs(ai): add acceptance test guide, skill, and justfile improvements Add docs/ai/acceptance-tests.md covering how to build dependencies and run acceptance tests locally. Add a run-acceptance-tests skill for AI agents. Refactor op-acceptance-tests/justfile to extract build-deps target and add a lightweight `just test` target for running individual tests with automatic dependency building. Remove git submodule update from local builds to avoid reverting in-progress submodule work. Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude/skills/run-acceptance-tests/SKILL.md | 85 +++++++++++++++++ AGENTS.md | 1 + docs/ai/acceptance-tests.md | 99 ++++++++++++++++++++ docs/ai/dev-workflow.md | 2 +- op-acceptance-tests/justfile | 62 ++++++------ 5 files changed, 219 insertions(+), 30 deletions(-) create mode 100644 .claude/skills/run-acceptance-tests/SKILL.md create mode 100644 docs/ai/acceptance-tests.md diff --git a/.claude/skills/run-acceptance-tests/SKILL.md b/.claude/skills/run-acceptance-tests/SKILL.md new file mode 100644 index 0000000000000..6d2682dc2a9ba --- /dev/null +++ b/.claude/skills/run-acceptance-tests/SKILL.md @@ -0,0 +1,85 @@ +--- +name: run-acceptance-tests +description: Build dependencies and run acceptance tests locally in the Optimism monorepo. Handles contracts, cannon prestates, Rust binaries, and optional kona prestate. +--- + +# Run Acceptance Tests + +Build all dependencies and run acceptance tests locally against the in-process devnet. + +See `docs/ai/acceptance-tests.md` for full reference. + +## When to Use + +- Before merging changes that affect devnet behavior +- When CI acceptance tests fail and you need to reproduce locally +- When asked to verify a change works end-to-end + +## Prerequisites + +- The optimism monorepo checked out with submodules initialized +- mise activated in the shell (`eval "$(mise activate bash)"` if not already active) and `mise install` completed (provides `just`, `gotestsum`, `forge`, etc.) +- A working C toolchain (`clang` or `gcc`) for Rust builds +- For kona reproducible prestates: Docker must be available + +## Steps + +### 1. Run Tests + +All `just` targets automatically build dependencies (contracts, cannon prestates, Rust binaries) before running. Always set `RUST_JIT_BUILD=1` so the test framework automatically builds any additional Rust binaries it needs (e.g. op-reth). Run from `op-acceptance-tests/`: + +**Specific test:** +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just test -run TestMyTest ./op-acceptance-tests/tests/base/ +``` + +**Specific package:** +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just test ./op-acceptance-tests/tests/base/... +``` + +**All tests:** +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test +``` + +**Gated subset** (e.g. `base`): +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test base +``` + +### 2. Kona Reproducible Prestate (if needed) + +Only for superfaultproofs / interop fault proof tests. Not handled by `build-deps` or `RUST_JIT_BUILD`: +```bash +just reproducible-prestate-kona +``` +**Requires Docker.** If Docker is not available, ask the user to run this command manually. + +### 3. Interpret Results + +When using `just acceptance-test`, structured output is available: +- Logs in `op-acceptance-tests/logs/testrun-/` +- JUnit XML in `op-acceptance-tests/results/results.xml` +- `flaky-tests.txt` in the log directory lists tests marked with `MarkFlaky()` + +When using `just test`, output goes to stdout only. + +## Tuning (`acceptance-test` only) + +When using `just acceptance-test`, override parallelism with environment variables: +- `ACCEPTANCE_TEST_PARALLEL=2` — limit per-package test parallelism +- `ACCEPTANCE_TEST_JOBS=4` — limit concurrent packages +- `ACCEPTANCE_TEST_TIMEOUT=1h` — per-package timeout +- `LOG_LEVEL=debug` — increase log verbosity + +## Troubleshooting + +| Symptom | Fix | +|---|---| +| Missing `prestate-mt64.bin.gz` | `cd op-acceptance-tests && just build-deps` | +| ABI mismatch / contract errors | `cd op-acceptance-tests && just build-deps` | +| `kona-node` / `op-rbuilder` / `rollup-boost` not found | `cd op-acceptance-tests && just build-deps` | +| `gotestsum` not found | `mise install` | +| `op-reth` binary not available | Ensure `RUST_JIT_BUILD=1` is set | +| Kona prestate hash mismatch | `just reproducible-prestate-kona` (requires Docker) | diff --git a/AGENTS.md b/AGENTS.md index 95abd2a6c7c7f..08404a837113a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,6 +68,7 @@ More detailed guidance for AI agents can be found in: - [docs/ai/contract-dev.md](docs/ai/contract-dev.md) - Smart contract development - [docs/ai/go-dev.md](docs/ai/go-dev.md) - Go service development - [docs/ai/rust-dev.md](docs/ai/rust-dev.md) - Rust development (kona, op-reth, alloy crates) +- [docs/ai/acceptance-tests.md](docs/ai/acceptance-tests.md) - Building and running acceptance tests locally ## External References diff --git a/docs/ai/acceptance-tests.md b/docs/ai/acceptance-tests.md new file mode 100644 index 0000000000000..4261da7b81a64 --- /dev/null +++ b/docs/ai/acceptance-tests.md @@ -0,0 +1,99 @@ +# Acceptance Tests + +Guidance for AI agents building and running acceptance tests in the Optimism monorepo. See [dev-workflow.md](dev-workflow.md) for tool versions and general workflow. + +## What Are Acceptance Tests? + +Acceptance tests live in `op-acceptance-tests/tests/` and run full in-process devnet scenarios. They exercise the entire stack — contracts, Go services, and Rust binaries — in a single `go test` process. This means all dependencies must be built locally before running them. + +## Running Tests + +All `just` targets below automatically build dependencies (contracts, cannon prestates, Rust binaries) before running tests. The builds are incremental — re-running is fast when nothing changed. + +**Prerequisites:** mise must be activated in the shell (see [dev-workflow.md](dev-workflow.md)), and a working C toolchain (`clang` or `gcc`) must be available for Rust builds. + +Always set `RUST_JIT_BUILD=1` when running locally. This lets the test framework automatically build any Rust binaries it needs (e.g. op-reth) using cargo's rebuild detection, so you don't have to pre-build them separately. + +Run from `op-acceptance-tests/`: + +### Specific Tests or Packages (recommended) + +```bash +# Run a single test +RUST_JIT_BUILD=1 cd op-acceptance-tests && just test -run TestMyTest ./op-acceptance-tests/tests/base/ + +# Run a package +RUST_JIT_BUILD=1 cd op-acceptance-tests && just test ./op-acceptance-tests/tests/base/... +``` + +The `just test` target builds deps, then runs `go test -count=1 -timeout 30m` with your arguments. + +### All Tests + +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test +``` + +Runs all test packages with gotestsum, structured logging, and auto-tuned parallelism. + +### Gated Subsets + +Gate files in `op-acceptance-tests/gates/` list package subsets: + +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test base +``` + +This runs only packages listed in `gates/base.txt`. + +### Kona Reproducible Prestate + +Some tests (e.g. superfaultproofs, interop fault proofs) require a reproducible kona prestate. This is **not** handled by `build-deps` or `RUST_JIT_BUILD`: + +```bash +just reproducible-prestate-kona +``` + +**Requires Docker.** If Docker is not available in your environment, ask the user to run this command for you. + +## What `build-deps` Does + +The `just build-deps` target (called automatically by `just test` and `just acceptance-test`) runs these steps when not in CI: + +1. **mise** — `mise install` (ensures gotestsum, forge, etc. are available) +2. **Contracts** — `cd packages/contracts-bedrock && just install && just build-no-tests` +3. **Cannon prestates** — `just cannon-prestates` (builds cannon, op-program, and prestate artifacts) +4. **Rust binaries** — `just build-rust-release` (kona-node, op-rbuilder, rollup-boost) + +You can also run `just build-deps` directly to pre-build without running tests. + +## Tuning Parallelism (`acceptance-test` only) + +When using `just acceptance-test`, the runner auto-detects CPU count and sets: +- `ACCEPTANCE_TEST_JOBS` — number of packages to test in parallel (default: CPU count) +- `ACCEPTANCE_TEST_PARALLEL` — `go test -parallel` value per package (default: CPU count / 2) +- `ACCEPTANCE_TEST_TIMEOUT` — per-package timeout (default: 2h) + +Override with environment variables: + +```bash +ACCEPTANCE_TEST_PARALLEL=2 ACCEPTANCE_TEST_TIMEOUT=1h cd op-acceptance-tests && just acceptance-test +``` + +## Log Output (`acceptance-test` only) + +When using `just acceptance-test`, logs are written to `op-acceptance-tests/logs/testrun-/`: +- `all.log` — full test output +- `raw_go_events.log` — JSON test events +- `flaky-tests.txt` — tests marked with `MarkFlaky()` + +Results XML goes to `op-acceptance-tests/results/results.xml`. + +When using `just test`, output goes to stdout only. + +## Common Issues + +- **Missing prestates** — Run `cd op-acceptance-tests && just build-deps` or `just cannon-prestates` from the repo root. +- **Stale contracts** — Rebuild with `cd packages/contracts-bedrock && just build-no-tests`. +- **Missing Rust binaries** — Run `just build-rust-release` from the repo root. +- **gotestsum not found** — Run `mise install` to install all pinned tools. diff --git a/docs/ai/dev-workflow.md b/docs/ai/dev-workflow.md index c2b6005fd1fc1..716d4df756ddf 100644 --- a/docs/ai/dev-workflow.md +++ b/docs/ai/dev-workflow.md @@ -1,6 +1,6 @@ # Development Workflow -Common workflow guidance for AI agents working in the Optimism monorepo. Language-specific details are in [go-dev.md](go-dev.md) and [rust-dev.md](rust-dev.md). +Common workflow guidance for AI agents working in the Optimism monorepo. Language-specific details are in [go-dev.md](go-dev.md) and [rust-dev.md](rust-dev.md). For end-to-end testing, see [acceptance-tests.md](acceptance-tests.md). ## Tool Versions diff --git a/op-acceptance-tests/justfile b/op-acceptance-tests/justfile index 7b68fa982dc2e..d06c358a30b90 100644 --- a/op-acceptance-tests/justfile +++ b/op-acceptance-tests/justfile @@ -4,8 +4,40 @@ REPO_ROOT := `realpath ..` # path to the root of the optimism monorepo default: @just acceptance-test +# Build all dependencies for local acceptance test runs. Skipped in CI where jobs handle this. +build-deps: + #!/usr/bin/env bash + set -euo pipefail + if [[ -n "${CIRCLECI:-}" ]]; then exit 0; fi + + echo "Building contracts (local build)..." + cd {{REPO_ROOT}} + echo " - Installing mise..." + mise install + cd packages/contracts-bedrock + echo " - Installing contracts..." + just install + echo " - Forge build..." + just build-no-tests + cd {{REPO_ROOT}} + + echo "Building cannon dependencies..." + cd {{REPO_ROOT}} + just cannon-prestates + + echo "Building Rust binaries (kona-node, op-rbuilder, rollup-boost)..." + cd {{REPO_ROOT}} + just build-rust-release + +# Run specific acceptance tests. Builds dependencies first, then passes all arguments to go test. +# Examples: +# just test ./op-acceptance-tests/tests/base/... +# just test -run TestMyTest ./op-acceptance-tests/tests/base/ +test *args: build-deps + cd {{REPO_ROOT}} && go test -count=1 -timeout 30m {{args}} + # Run acceptance tests, optionally filtered by a gate (reads packages from gates/.txt) -acceptance-test gate="": +acceptance-test gate="": build-deps #!/usr/bin/env bash set -euo pipefail GATE="{{gate}}" @@ -15,34 +47,6 @@ acceptance-test gate="": echo -e "DEVNET: in-memory, MODE: all tests\n" fi - # Build dependencies for in-process mode if not in CI. - # In CI jobs already take care of this, so we skip it. - if [[ -z "${CIRCLECI:-}" ]]; then - echo "Building contracts (local build)..." - cd {{REPO_ROOT}} - echo " - Updating submodules..." - git submodule update --init --recursive --single-branch -j 8 - echo " - Installing mise..." - mise install - cd packages/contracts-bedrock - echo " - Installing contracts..." - just install - echo " - Forge build..." - just build-no-tests - cd {{REPO_ROOT}} - - echo "Checking cannon dependencies..." - if [ ! -e {{REPO_ROOT}}/cannon/bin/cannon ] || [ ! -e {{REPO_ROOT}}/op-program/bin/prestate-mt64.bin.gz ]; then - echo "Building cannon dependencies..." - cd {{REPO_ROOT}} - make cannon-prestates - fi - - echo "Building Rust binaries (kona-node, op-rbuilder, rollup-boost)..." - cd {{REPO_ROOT}} - just build-rust-release - fi - LOG_LEVEL="$(echo "${LOG_LEVEL:-info}" | grep -E '^(debug|info|warn|error)$' || echo 'info')" echo "LOG_LEVEL: $LOG_LEVEL" From 6e78849456b7c538a08d1352fb96e30cdbecedf3 Mon Sep 17 00:00:00 2001 From: Adrian Sutton Date: Thu, 2 Apr 2026 21:33:14 +0000 Subject: [PATCH 2/2] =?UTF-8?q?fix(ai):=20address=20PR=20review=20?= =?UTF-8?q?=E2=80=94=20deduplicate=20skill/doc,=20add=20mise=20exec=20pref?= =?UTF-8?q?ix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Slim down the acceptance test skill to a thin wrapper that @-includes the doc. Add `mise exec --` prefix to all just commands since AI agent shells don't have mise activated. Document mise setup in dev-workflow.md. Co-Authored-By: Claude Opus 4.6 (1M context) --- .claude/skills/run-acceptance-tests/SKILL.md | 73 +------------------- docs/ai/acceptance-tests.md | 20 +++--- docs/ai/dev-workflow.md | 8 +++ 3 files changed, 20 insertions(+), 81 deletions(-) diff --git a/.claude/skills/run-acceptance-tests/SKILL.md b/.claude/skills/run-acceptance-tests/SKILL.md index 6d2682dc2a9ba..74a509b887c4e 100644 --- a/.claude/skills/run-acceptance-tests/SKILL.md +++ b/.claude/skills/run-acceptance-tests/SKILL.md @@ -5,81 +5,12 @@ description: Build dependencies and run acceptance tests locally in the Optimism # Run Acceptance Tests -Build all dependencies and run acceptance tests locally against the in-process devnet. - -See `docs/ai/acceptance-tests.md` for full reference. - ## When to Use - Before merging changes that affect devnet behavior - When CI acceptance tests fail and you need to reproduce locally - When asked to verify a change works end-to-end -## Prerequisites - -- The optimism monorepo checked out with submodules initialized -- mise activated in the shell (`eval "$(mise activate bash)"` if not already active) and `mise install` completed (provides `just`, `gotestsum`, `forge`, etc.) -- A working C toolchain (`clang` or `gcc`) for Rust builds -- For kona reproducible prestates: Docker must be available - -## Steps - -### 1. Run Tests - -All `just` targets automatically build dependencies (contracts, cannon prestates, Rust binaries) before running. Always set `RUST_JIT_BUILD=1` so the test framework automatically builds any additional Rust binaries it needs (e.g. op-reth). Run from `op-acceptance-tests/`: - -**Specific test:** -```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just test -run TestMyTest ./op-acceptance-tests/tests/base/ -``` - -**Specific package:** -```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just test ./op-acceptance-tests/tests/base/... -``` - -**All tests:** -```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test -``` - -**Gated subset** (e.g. `base`): -```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test base -``` - -### 2. Kona Reproducible Prestate (if needed) - -Only for superfaultproofs / interop fault proof tests. Not handled by `build-deps` or `RUST_JIT_BUILD`: -```bash -just reproducible-prestate-kona -``` -**Requires Docker.** If Docker is not available, ask the user to run this command manually. - -### 3. Interpret Results - -When using `just acceptance-test`, structured output is available: -- Logs in `op-acceptance-tests/logs/testrun-/` -- JUnit XML in `op-acceptance-tests/results/results.xml` -- `flaky-tests.txt` in the log directory lists tests marked with `MarkFlaky()` - -When using `just test`, output goes to stdout only. - -## Tuning (`acceptance-test` only) - -When using `just acceptance-test`, override parallelism with environment variables: -- `ACCEPTANCE_TEST_PARALLEL=2` — limit per-package test parallelism -- `ACCEPTANCE_TEST_JOBS=4` — limit concurrent packages -- `ACCEPTANCE_TEST_TIMEOUT=1h` — per-package timeout -- `LOG_LEVEL=debug` — increase log verbosity - -## Troubleshooting +## Guide -| Symptom | Fix | -|---|---| -| Missing `prestate-mt64.bin.gz` | `cd op-acceptance-tests && just build-deps` | -| ABI mismatch / contract errors | `cd op-acceptance-tests && just build-deps` | -| `kona-node` / `op-rbuilder` / `rollup-boost` not found | `cd op-acceptance-tests && just build-deps` | -| `gotestsum` not found | `mise install` | -| `op-reth` binary not available | Ensure `RUST_JIT_BUILD=1` is set | -| Kona prestate hash mismatch | `just reproducible-prestate-kona` (requires Docker) | +@docs/ai/acceptance-tests.md diff --git a/docs/ai/acceptance-tests.md b/docs/ai/acceptance-tests.md index 4261da7b81a64..2b91777e464c5 100644 --- a/docs/ai/acceptance-tests.md +++ b/docs/ai/acceptance-tests.md @@ -10,7 +10,7 @@ Acceptance tests live in `op-acceptance-tests/tests/` and run full in-process de All `just` targets below automatically build dependencies (contracts, cannon prestates, Rust binaries) before running tests. The builds are incremental — re-running is fast when nothing changed. -**Prerequisites:** mise must be activated in the shell (see [dev-workflow.md](dev-workflow.md)), and a working C toolchain (`clang` or `gcc`) must be available for Rust builds. +**Prerequisites:** mise tools must be installed (see [dev-workflow.md](dev-workflow.md#setup)), and a working C toolchain (`clang` or `gcc`) must be available for Rust builds. Always set `RUST_JIT_BUILD=1` when running locally. This lets the test framework automatically build any Rust binaries it needs (e.g. op-reth) using cargo's rebuild detection, so you don't have to pre-build them separately. @@ -20,10 +20,10 @@ Run from `op-acceptance-tests/`: ```bash # Run a single test -RUST_JIT_BUILD=1 cd op-acceptance-tests && just test -run TestMyTest ./op-acceptance-tests/tests/base/ +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just test -run TestMyTest ./op-acceptance-tests/tests/base/ # Run a package -RUST_JIT_BUILD=1 cd op-acceptance-tests && just test ./op-acceptance-tests/tests/base/... +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just test ./op-acceptance-tests/tests/base/... ``` The `just test` target builds deps, then runs `go test -count=1 -timeout 30m` with your arguments. @@ -31,7 +31,7 @@ The `just test` target builds deps, then runs `go test -count=1 -timeout 30m` wi ### All Tests ```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just acceptance-test ``` Runs all test packages with gotestsum, structured logging, and auto-tuned parallelism. @@ -41,7 +41,7 @@ Runs all test packages with gotestsum, structured logging, and auto-tuned parall Gate files in `op-acceptance-tests/gates/` list package subsets: ```bash -RUST_JIT_BUILD=1 cd op-acceptance-tests && just acceptance-test base +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just acceptance-test base ``` This runs only packages listed in `gates/base.txt`. @@ -51,7 +51,7 @@ This runs only packages listed in `gates/base.txt`. Some tests (e.g. superfaultproofs, interop fault proofs) require a reproducible kona prestate. This is **not** handled by `build-deps` or `RUST_JIT_BUILD`: ```bash -just reproducible-prestate-kona +mise exec -- just reproducible-prestate-kona ``` **Requires Docker.** If Docker is not available in your environment, ask the user to run this command for you. @@ -77,7 +77,7 @@ When using `just acceptance-test`, the runner auto-detects CPU count and sets: Override with environment variables: ```bash -ACCEPTANCE_TEST_PARALLEL=2 ACCEPTANCE_TEST_TIMEOUT=1h cd op-acceptance-tests && just acceptance-test +ACCEPTANCE_TEST_PARALLEL=2 ACCEPTANCE_TEST_TIMEOUT=1h cd op-acceptance-tests && mise exec -- just acceptance-test ``` ## Log Output (`acceptance-test` only) @@ -93,7 +93,7 @@ When using `just test`, output goes to stdout only. ## Common Issues -- **Missing prestates** — Run `cd op-acceptance-tests && just build-deps` or `just cannon-prestates` from the repo root. -- **Stale contracts** — Rebuild with `cd packages/contracts-bedrock && just build-no-tests`. -- **Missing Rust binaries** — Run `just build-rust-release` from the repo root. +- **Missing prestates** — Run `cd op-acceptance-tests && mise exec -- just build-deps` or `mise exec -- just cannon-prestates` from the repo root. +- **Stale contracts** — Rebuild with `cd packages/contracts-bedrock && mise exec -- just build-no-tests`. +- **Missing Rust binaries** — Run `mise exec -- just build-rust-release` from the repo root. - **gotestsum not found** — Run `mise install` to install all pinned tools. diff --git a/docs/ai/dev-workflow.md b/docs/ai/dev-workflow.md index 716d4df756ddf..421b0e39c6aa9 100644 --- a/docs/ai/dev-workflow.md +++ b/docs/ai/dev-workflow.md @@ -8,6 +8,14 @@ All tool versions are pinned in `mise.toml` at the repo root. Always access tool If mise reports the repo isn't trusted, ask the user to run `mise trust` — never trust it automatically. +### Setup + +Run `mise install` to install all pinned tools (just, gotestsum, forge, etc.). AI agent shells typically do not have mise activated, so prefix commands with `mise exec --` to ensure tools are on `PATH`: + +```bash +mise exec -- just +``` + ## Build System The repo uses [Just](https://github.com/casey/just) as its build system. Shared justfile infrastructure lives in `justfiles/`. Each component has its own justfile — run `just --list` in any directory to see available targets.