diff --git a/.claude/skills/run-acceptance-tests/SKILL.md b/.claude/skills/run-acceptance-tests/SKILL.md new file mode 100644 index 0000000000000..74a509b887c4e --- /dev/null +++ b/.claude/skills/run-acceptance-tests/SKILL.md @@ -0,0 +1,16 @@ +--- +name: run-acceptance-tests +description: Build dependencies and run acceptance tests locally in the Optimism monorepo. Handles contracts, cannon prestates, Rust binaries, and optional kona prestate. +--- + +# Run Acceptance Tests + +## When to Use + +- Before merging changes that affect devnet behavior +- When CI acceptance tests fail and you need to reproduce locally +- When asked to verify a change works end-to-end + +## Guide + +@docs/ai/acceptance-tests.md diff --git a/AGENTS.md b/AGENTS.md index 95abd2a6c7c7f..08404a837113a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,6 +68,7 @@ More detailed guidance for AI agents can be found in: - [docs/ai/contract-dev.md](docs/ai/contract-dev.md) - Smart contract development - [docs/ai/go-dev.md](docs/ai/go-dev.md) - Go service development - [docs/ai/rust-dev.md](docs/ai/rust-dev.md) - Rust development (kona, op-reth, alloy crates) +- [docs/ai/acceptance-tests.md](docs/ai/acceptance-tests.md) - Building and running acceptance tests locally ## External References diff --git a/docs/ai/acceptance-tests.md b/docs/ai/acceptance-tests.md new file mode 100644 index 0000000000000..2b91777e464c5 --- /dev/null +++ b/docs/ai/acceptance-tests.md @@ -0,0 +1,99 @@ +# Acceptance Tests + +Guidance for AI agents building and running acceptance tests in the Optimism monorepo. See [dev-workflow.md](dev-workflow.md) for tool versions and general workflow. + +## What Are Acceptance Tests? + +Acceptance tests live in `op-acceptance-tests/tests/` and run full in-process devnet scenarios. They exercise the entire stack — contracts, Go services, and Rust binaries — in a single `go test` process. This means all dependencies must be built locally before running them. + +## Running Tests + +All `just` targets below automatically build dependencies (contracts, cannon prestates, Rust binaries) before running tests. The builds are incremental — re-running is fast when nothing changed. + +**Prerequisites:** mise tools must be installed (see [dev-workflow.md](dev-workflow.md#setup)), and a working C toolchain (`clang` or `gcc`) must be available for Rust builds. + +Always set `RUST_JIT_BUILD=1` when running locally. This lets the test framework automatically build any Rust binaries it needs (e.g. op-reth) using cargo's rebuild detection, so you don't have to pre-build them separately. + +Run from `op-acceptance-tests/`: + +### Specific Tests or Packages (recommended) + +```bash +# Run a single test +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just test -run TestMyTest ./op-acceptance-tests/tests/base/ + +# Run a package +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just test ./op-acceptance-tests/tests/base/... +``` + +The `just test` target builds deps, then runs `go test -count=1 -timeout 30m` with your arguments. + +### All Tests + +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just acceptance-test +``` + +Runs all test packages with gotestsum, structured logging, and auto-tuned parallelism. + +### Gated Subsets + +Gate files in `op-acceptance-tests/gates/` list package subsets: + +```bash +RUST_JIT_BUILD=1 cd op-acceptance-tests && mise exec -- just acceptance-test base +``` + +This runs only packages listed in `gates/base.txt`. + +### Kona Reproducible Prestate + +Some tests (e.g. superfaultproofs, interop fault proofs) require a reproducible kona prestate. This is **not** handled by `build-deps` or `RUST_JIT_BUILD`: + +```bash +mise exec -- just reproducible-prestate-kona +``` + +**Requires Docker.** If Docker is not available in your environment, ask the user to run this command for you. + +## What `build-deps` Does + +The `just build-deps` target (called automatically by `just test` and `just acceptance-test`) runs these steps when not in CI: + +1. **mise** — `mise install` (ensures gotestsum, forge, etc. are available) +2. **Contracts** — `cd packages/contracts-bedrock && just install && just build-no-tests` +3. **Cannon prestates** — `just cannon-prestates` (builds cannon, op-program, and prestate artifacts) +4. **Rust binaries** — `just build-rust-release` (kona-node, op-rbuilder, rollup-boost) + +You can also run `just build-deps` directly to pre-build without running tests. + +## Tuning Parallelism (`acceptance-test` only) + +When using `just acceptance-test`, the runner auto-detects CPU count and sets: +- `ACCEPTANCE_TEST_JOBS` — number of packages to test in parallel (default: CPU count) +- `ACCEPTANCE_TEST_PARALLEL` — `go test -parallel` value per package (default: CPU count / 2) +- `ACCEPTANCE_TEST_TIMEOUT` — per-package timeout (default: 2h) + +Override with environment variables: + +```bash +ACCEPTANCE_TEST_PARALLEL=2 ACCEPTANCE_TEST_TIMEOUT=1h cd op-acceptance-tests && mise exec -- just acceptance-test +``` + +## Log Output (`acceptance-test` only) + +When using `just acceptance-test`, logs are written to `op-acceptance-tests/logs/testrun-/`: +- `all.log` — full test output +- `raw_go_events.log` — JSON test events +- `flaky-tests.txt` — tests marked with `MarkFlaky()` + +Results XML goes to `op-acceptance-tests/results/results.xml`. + +When using `just test`, output goes to stdout only. + +## Common Issues + +- **Missing prestates** — Run `cd op-acceptance-tests && mise exec -- just build-deps` or `mise exec -- just cannon-prestates` from the repo root. +- **Stale contracts** — Rebuild with `cd packages/contracts-bedrock && mise exec -- just build-no-tests`. +- **Missing Rust binaries** — Run `mise exec -- just build-rust-release` from the repo root. +- **gotestsum not found** — Run `mise install` to install all pinned tools. diff --git a/docs/ai/dev-workflow.md b/docs/ai/dev-workflow.md index c2b6005fd1fc1..421b0e39c6aa9 100644 --- a/docs/ai/dev-workflow.md +++ b/docs/ai/dev-workflow.md @@ -1,6 +1,6 @@ # Development Workflow -Common workflow guidance for AI agents working in the Optimism monorepo. Language-specific details are in [go-dev.md](go-dev.md) and [rust-dev.md](rust-dev.md). +Common workflow guidance for AI agents working in the Optimism monorepo. Language-specific details are in [go-dev.md](go-dev.md) and [rust-dev.md](rust-dev.md). For end-to-end testing, see [acceptance-tests.md](acceptance-tests.md). ## Tool Versions @@ -8,6 +8,14 @@ All tool versions are pinned in `mise.toml` at the repo root. Always access tool If mise reports the repo isn't trusted, ask the user to run `mise trust` — never trust it automatically. +### Setup + +Run `mise install` to install all pinned tools (just, gotestsum, forge, etc.). AI agent shells typically do not have mise activated, so prefix commands with `mise exec --` to ensure tools are on `PATH`: + +```bash +mise exec -- just +``` + ## Build System The repo uses [Just](https://github.com/casey/just) as its build system. Shared justfile infrastructure lives in `justfiles/`. Each component has its own justfile — run `just --list` in any directory to see available targets. diff --git a/op-acceptance-tests/justfile b/op-acceptance-tests/justfile index 7b68fa982dc2e..d06c358a30b90 100644 --- a/op-acceptance-tests/justfile +++ b/op-acceptance-tests/justfile @@ -4,8 +4,40 @@ REPO_ROOT := `realpath ..` # path to the root of the optimism monorepo default: @just acceptance-test +# Build all dependencies for local acceptance test runs. Skipped in CI where jobs handle this. +build-deps: + #!/usr/bin/env bash + set -euo pipefail + if [[ -n "${CIRCLECI:-}" ]]; then exit 0; fi + + echo "Building contracts (local build)..." + cd {{REPO_ROOT}} + echo " - Installing mise..." + mise install + cd packages/contracts-bedrock + echo " - Installing contracts..." + just install + echo " - Forge build..." + just build-no-tests + cd {{REPO_ROOT}} + + echo "Building cannon dependencies..." + cd {{REPO_ROOT}} + just cannon-prestates + + echo "Building Rust binaries (kona-node, op-rbuilder, rollup-boost)..." + cd {{REPO_ROOT}} + just build-rust-release + +# Run specific acceptance tests. Builds dependencies first, then passes all arguments to go test. +# Examples: +# just test ./op-acceptance-tests/tests/base/... +# just test -run TestMyTest ./op-acceptance-tests/tests/base/ +test *args: build-deps + cd {{REPO_ROOT}} && go test -count=1 -timeout 30m {{args}} + # Run acceptance tests, optionally filtered by a gate (reads packages from gates/.txt) -acceptance-test gate="": +acceptance-test gate="": build-deps #!/usr/bin/env bash set -euo pipefail GATE="{{gate}}" @@ -15,34 +47,6 @@ acceptance-test gate="": echo -e "DEVNET: in-memory, MODE: all tests\n" fi - # Build dependencies for in-process mode if not in CI. - # In CI jobs already take care of this, so we skip it. - if [[ -z "${CIRCLECI:-}" ]]; then - echo "Building contracts (local build)..." - cd {{REPO_ROOT}} - echo " - Updating submodules..." - git submodule update --init --recursive --single-branch -j 8 - echo " - Installing mise..." - mise install - cd packages/contracts-bedrock - echo " - Installing contracts..." - just install - echo " - Forge build..." - just build-no-tests - cd {{REPO_ROOT}} - - echo "Checking cannon dependencies..." - if [ ! -e {{REPO_ROOT}}/cannon/bin/cannon ] || [ ! -e {{REPO_ROOT}}/op-program/bin/prestate-mt64.bin.gz ]; then - echo "Building cannon dependencies..." - cd {{REPO_ROOT}} - make cannon-prestates - fi - - echo "Building Rust binaries (kona-node, op-rbuilder, rollup-boost)..." - cd {{REPO_ROOT}} - just build-rust-release - fi - LOG_LEVEL="$(echo "${LOG_LEVEL:-info}" | grep -E '^(debug|info|warn|error)$' || echo 'info')" echo "LOG_LEVEL: $LOG_LEVEL"