feat(l2): merge ZK-DEX e2e and CI fixes into app-customized-framework#12
Merged
Zena-park merged 107 commits intofeat/app-customized-frameworkfrom Mar 4, 2026
Merged
Conversation
Add DexCircuit that processes ERC-20 style token transfers using verified storage proofs instead of the previous keccak-chaining demo. Includes Solidity-compatible storage layout, ABI parsing, and 15 tests.
…ncremental MPT) Introduce the common infrastructure for app-specific guest programs: - AppCircuit trait with classify_tx/execute_operation/gas_cost/generate_logs - AppState for in-circuit state management via storage proofs - AppProgramInput/StorageProof/AccountProof types - Incremental MPT for state root recomputation from proof deltas
…ries Add cycle-tracker-report start/end prints around read_input, execution, and commit_public_inputs phases for performance profiling.
…arison CLI binary that measures cycles and proving time for zk-dex, tokamon, and evm-l2 using SP1 SDK. Supports --execute-only and --format groth16.
Add guest_program_id config field to ProofCoordinatorConfig and replace hardcoded "evm-l2" string. Expose --proof-coordinator.guest-program-id CLI arg and ETHREX_GUEST_PROGRAM_ID env var.
…verride Add runtime configuration for selecting guest programs: programs.toml with enabled_programs list, docker-compose override for injecting ETHREX_GUEST_PROGRAM_ID, and Makefile targets for guest program mode.
Add dedicated Launch L2 flow page and setup guide. Rename "Deployments" to "My L2s" across navigation, dashboard, and detail views. Replace inline modals with /launch?program= links.
Add seedOfficialPrograms() at DB init to insert evm-l2, zk-dex, and tokamon entries. JOIN programs table in getDeploymentById for metadata. Accept config field in POST /api/deployments.
…circuit design Add end-user GP Store deployment guide and developer notes covering L2 launch platform design, SP1 performance comparison, and app-specific circuit architecture and development plan.
Ignore Next.js build output, SP1 guest ELF/VK binaries, SP1 guest Cargo.lock files, SQLite WAL/SHM files, and generated L2 config.
Migrate the zk-dex guest program from the old keccak-chaining demo (DexProgramInput/execution_program) to the new AppCircuit-based architecture (AppProgramInput/execute_app_circuit with DexCircuit). - Rewrite bin/sp1-zk-dex to use execute_app_circuit(DexCircuit, input) - Add rkyv derives to AppProgramInput, StorageProof, AccountProof - Add ethrex-common dep and secp256k1 SP1 patch to sp1-zk-dex - Delete old execution.rs and types.rs from zk_dex module - Update sp1_benchmark.rs zk-dex input gen to TODO stub (Phase 3) - Replace registry test with DexCircuit-based unit test
Implement prover-side input conversion so the zk-dex guest program receives AppProgramInput (Merkle proofs) from the standard ProgramInput (full ExecutionWitness trie), eliminating the need for coordinator or protocol changes. - Add input_converter module with convert_to_app_input() that rebuilds state/storage tries from ExecutionWitness and extracts Merkle proofs - Replace ZkDexGuestProgram::serialize_input() pass-through with ProgramInput deserialization, transaction analysis, and conversion - Add analyze_zk_dex_transactions() to extract needed accounts and storage slots from batch transactions - Implement generate_zk_dex_input() in sp1_benchmark exercising the full conversion pipeline with mock state tries and transfer txs - Update tests in traits.rs and registry.rs for the new behavior
- Fix build_trie_from_proofs storing intermediate nodes at hash-based paths instead of nibble paths, breaking multi-account trie lookups - Add multi-account test coverage for incremental MPT operations - Implement EIP-1559 transaction signing in SP1 benchmark using deterministic secp256k1 keys - Generate AppProgramInput directly with Merkle proofs from trie - Feature-gate serialize_input for zkVM guest (no l2 feature) - Fix Address type alias usage in SP1 zk-dex guest binary
Add missing SP1 patched crates (k256, ecdsa, crypto-bigint, sha2, sha3, p256) to the zk-dex guest binary. This enables SP1 precompiles for ECDSA signature recovery and hashing, reducing execution cycles from 11.4M to 357K (182x vs EVM L2 baseline) and proving time from 5m05s to 3m26s. Update benchmark comparison document with 3-way results.
Migrate the zk-dex guest program from the old keccak-chaining demo (DexProgramInput/execution_program) to the new AppCircuit-based architecture (AppProgramInput/execute_app_circuit with DexCircuit). - Rewrite bin/sp1-zk-dex to use execute_app_circuit(DexCircuit, input) - Add rkyv derives to AppProgramInput, StorageProof, AccountProof - Add ethrex-common dep and secp256k1 SP1 patch to sp1-zk-dex - Delete old execution.rs and types.rs from zk_dex module - Update sp1_benchmark.rs zk-dex input gen to TODO stub (Phase 3) - Replace registry test with DexCircuit-based unit test
Implement prover-side input conversion so the zk-dex guest program receives AppProgramInput (Merkle proofs) from the standard ProgramInput (full ExecutionWitness trie), eliminating the need for coordinator or protocol changes. - Add input_converter module with convert_to_app_input() that rebuilds state/storage tries from ExecutionWitness and extracts Merkle proofs - Replace ZkDexGuestProgram::serialize_input() pass-through with ProgramInput deserialization, transaction analysis, and conversion - Add analyze_zk_dex_transactions() to extract needed accounts and storage slots from batch transactions - Implement generate_zk_dex_input() in sp1_benchmark exercising the full conversion pipeline with mock state tries and transfer txs - Update tests in traits.rs and registry.rs for the new behavior
Timeline, SP1 profiling results (182x cycle reduction), Halo2 vs SP1 decision, L1/L2 infra status, and next steps.
…tration Add deployer support for registering custom guest programs (e.g., zk-dex) on L1 during contract initialization, including GuestProgramRegistry registration and SP1 verification key setup via upgradeVerificationKey(). - Add --register-guest-programs and --zk-dex-sp1-vk CLI options - Add guest program registration loop in initialize_contracts() - Add resolve_deployer_program_type_id() and get_vk_for_program() helpers - Add ZK-DEX E2E Makefile targets (deploy, l2, prover) - Add programs-zk-dex.toml prover config - Add E2E design document
Three issues fixed: 1. Timelock ownership: VK registration called OnChainProposer directly but owner is Timelock. Added Timelock.upgradeVerificationKey() with SECURITY_COUNCIL role and updated deployer to use it. 2. KZG blob verification crash: kzg-rs BLS12-381 operations crash in SP1 VM despite sp1_bls12_381 dependency. App circuits now skip KZG verification (L1 already verifies) and compute versioned hash directly. 3. Empty batch safety: Added early returns in verify_state_proofs() and compute_new_state_root() when account proofs are empty. SP1 guest execution verified: 414K cycles (42K read + 370K exec + 2K commit).
Automates the 4-step ZK-DEX E2E environment setup (L1 start, contract deploy, L2 start, prover start) into a single script with start/stop/ status/logs commands and --no-prover option for app testing.
Increase health check timeouts to account for compilation time (L1: 30s -> 300s, L2: 60s -> 600s) and switch summary output from echo to printf to prevent garbled terminal formatting.
Add complete Docker infrastructure for running the ZK-DEX E2E pipeline with real SP1 ZK proofs and optional GPU acceleration. - Dockerfile.sp1: Multi-stage build with SP1 succinct toolchain, Docker CLI for Groth16 gnark wrapper, and ZK-DEX guest program compilation - docker-compose-zk-dex.overrides.yaml: SP1 prover with Docker-in-Docker volume path alignment for Groth16 proving step - docker-compose-zk-dex-gpu.overrides.yaml: NVIDIA GPU opt-in override - scripts/zk-dex-docker.sh: One-command Docker localnet management (start/stop/status/logs/clean with --no-gpu/--no-prover flags) - Makefile: 7 new Docker ZK-DEX targets - Verified: 203s proving time (3m 23s) on Apple M4 Max CPU-only Docker
…rification - Add withdraw-status.html: L2 withdrawal tracking page with 4-stage status view (TX confirmed → batch committed → batch verified → claimable), pagination (10/page), wallet-based filtering, auto-refresh, and block timestamp display - Add navigation links from dashboard and bridge UI to withdrawal tracker - Remove onlyOwner from verifyBatch() in OnChainProposer — ZK proof verification is cryptographically secured, making caller restriction unnecessary and enabling permissionless proving - Fix Dockerfile.sp1: copy SP1 guest program VK files to final image so the deployer can register them on L1 during contract initialization - Update bridge Dockerfile to include withdraw-status.html - Add 3D styling and color accents to dashboard quick link tiles
…ker tools
Three root causes of the 00e (Invalid SP1 proof) error:
1. Race condition: program_id was only stored on proof submission, but
the L1 committer needed it earlier to set the correct programTypeId.
Batches were committed as evm-l2 (type=1) while proofs were generated
for zk-dex (type=2), causing a VK mismatch.
Fix: Store program_id in proof_coordinator when batch is assigned to
prover, and fall back to ETHREX_GUEST_PROGRAM_ID env var in committer.
2. Custom program public values not implemented: The contract required
customPublicValues for programTypeId > 1, but the proof sender always
sent empty bytes. Since zk-dex uses the same ProgramOutput format as
evm-l2, we use _getPublicInputsFromCommitment() for all program types.
3. Dockerfile.sp1 missing mkdir for canonicalize() path resolution:
The deployer resolves VK paths via env!("CARGO_MANIFEST_DIR")/../../
which needs /ethrex/cmd/ethrex/ to exist in the final Docker image.
Also adds Docker tools infrastructure (Blockscout, Bridge UI, Dashboard)
with make targets and script commands for managing them.
Extract inline transaction handlers from app_execution.rs into dedicated modules under common/handlers/ to fix five root causes of SP1 proof verification failure: 1. Withdrawal now credits BURN_ADDRESS with withdrawn value 2. L1Messenger.lastMessageId storage slot incremented on withdrawal 3. Withdrawal logs (WithdrawalInitiated + L1Message) properly generated instead of empty vec, fixing l1_out_messages_merkle_root mismatch 4. Gas fee distribution now debits effective_gas_price (not just base_fee) and credits coinbase/base_fee_vault/operator_fee_vault 5. Witness analyzer includes BURN_ADDRESS, messenger storage, coinbase, and fee vault accounts for proof generation
…ee handlers Verify handlers produce EVM-identical state transitions and logs: - Withdrawal: sender debit, BURN_ADDRESS credit, messenger lastMessageId - Withdrawal logs: WithdrawalInitiated + L1Message with exact topic bytes - Integration: generated logs parseable by get_block_l1_messages() - Deposit: mintETH(0xb0f4d395) calldata parses real recipient correctly - Gas fees: effective_gas_price distribution to coinbase/vaults with conservation law (sender debit == sum of all credits)
…l witness inclusion - Use block.header.gas_used for per-tx gas computation instead of fixed constants (WITHDRAWAL_GAS=100k vs actual EVM gas=95,002), fixing the 00e SP1 proof verification failure on withdrawal/transfer batches - Only include withdrawal accounts (BURN_ADDRESS, L2_TO_L1_MESSENGER) and gas fee accounts (coinbase, fee vaults) in witness analysis when the batch actually contains those transaction types, preventing trie traversal errors on empty batches - Skip L1 commits for empty batches (no txs, no messages) after batch 1 E2E validated: all 3 batch types (empty, deposit, withdrawal+transfer) proved and verified on L1 without 00e errors.
Empty batches (only empty blocks, no transactions or messages) can now be verified on L1 without generating a ZK proof, saving ~3 minutes of proving time per empty batch. Changes: - OnChainProposer.sol: add _isEmptyBatch() check in verifyBatch() to skip SP1/RISC0/TDX proof verification for empty batches; exempt empty commits from blob requirement in commitBatch() - l1_committer.rs: fast-forward empty blocks in batch assembly without consuming blob space; send EIP-1559 (no blob) tx for empty batches - l1_proof_sender.rs: detect empty batches and send proof-free verifyBatch instead of waiting for prover - batch.rs: add Batch::is_empty_batch() helper with 7 unit tests - store.rs: expose get_non_privileged_transactions_by_batch() - docs/design/empty-batch-optimization.md: design document
- Use block header state_root directly for empty blocks instead of checkpoint store lookup (which didn't have empty blocks registered) - Register empty blocks in checkpoint store with empty account updates so forkchoice_update can find them - Skip witness generation for empty batches (seal_batch instead of seal_batch_with_prover_input) - Skip blob_gas_used metrics check for empty batch EIP-1559 commits - Fix false-positive empty batch detection in proof sender: check all 5 conditions (non_priv_tx, l1_in_hash, l1_out_msgs, balance_diffs, l2_in_msgs) instead of only non_privileged_transactions == 0
Add mint, spend, liquidate, convertNote, makeOrder, takeOrder, and settleOrder operations to the DexCircuit, matching the ZkDex Solidity contract's state transitions exactly. New files: - storage.rs: slot computation for notes, orders, encryptedNotes - events.rs: NoteStateChange, OrderTaken, OrderSettled log generation - notes.rs: mint/spend/liquidate/convertNote execution - orders.rs: makeOrder/takeOrder/settleOrder execution + RLP decoder Modified: - circuit.rs: ABI selectors, calldata parsing, routing for 8 ops - mod.rs: witness analyzer for all operation storage slots
Single app chain only needs Vanilla integration test. Re-enable TDX via vars.ENABLE_TDX_TEST = 'true'.
Enable via GitHub Settings > Variables: - ENABLE_VALIDIUM_TEST=true - ENABLE_WEB3SIGNER_TEST=true - ENABLE_BASED_TEST=true - ENABLE_TDX_TEST=true - ENABLE_SHARED_BRIDGE_TEST=true
Disabled on pull_request (kept workflow_dispatch for manual runs): - PR Status (lambdaclass project tracking) - PR Lines of Code Analysis - Benchmark Block execution - Benchmark LEVM vs REVM - Perf Changelog - L2 TDX build
When all matrix items are excluded (enabled: false), the job is not created. Changed gate job to only fail if job ran and failed, not when skipped.
GitHub Actions does not support `exclude` with `include`-only matrices. The exclude pattern caused the matrix job to fail silently, breaking the Integration Test L2 gate job. Comment out disabled variants instead.
…rehensive caching - Remove all --test-threads=1 flags (enable parallel test execution) - Add Rust build cache to integration-test-tdx and state-diff-test jobs - Remove duplicate cache entry in state-diff-test - Enable parallel tests in shared-bridge job (for future use) Expected speedup: - First run: 1h50m → ~50-60min (70% reduction via caching + parallelization) - Cached runs: ~20-30min (full cache hits) Context: - Vanilla-only matrix already active (75% fewer variants) - fail-fast: true already enabled in relevant jobs - Combined with previous caching optimizations
Docker Compose uses tokamak-app-* container names: - ethrex_l2 → tokamak-app-l2 - ethrex_l1 → tokamak-app-l1 Fixed in: - integration-test (Vanilla) - integration-test-tdx Existing optimizations: - Rust cache added (70% faster builds) - fail-fast enabled - --test-threads removed (parallel execution) - Vanilla only (other variants disabled)
Removed stray ]; then...fi block after commented shared-bridge check. This was causing: 'did not find expected key' YAML parse error at line 999. Fixes job 65691595733
Add "Invalid Missing Ancestor Syncing ReOrg, Transaction Value, EmptyTxs=False, CanonicalReOrg=True, Invalid P9" to the known-flaky tests list. This is the same test family already allowlisted but with the CanonicalReOrg=True variant, which causes intermittent CI failures due to hive framework timing issues.
feat(l2): add ZK-DEX circuit, E2E infrastructure, and Docker polish
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly advances the modularity and efficiency of L2 operations by introducing a new app-specific circuit execution framework. This framework allows specialized guest programs to perform state transitions in native Rust, bypassing the overhead of a full EVM interpreter. The ZK-DEX guest program has been refactored to utilize this new architecture, promising substantial performance gains in proving times. Additionally, optimizations for empty batches have been implemented to reduce unnecessary ZK proof generation and L1 gas costs, alongside various Docker and CI improvements to streamline development and testing workflows. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Changelog
Ignored Files
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a significant and well-designed architectural change to support modular guest programs, moving away from a monolithic EVM execution model for specialized L2s. The implementation of the AppCircuit framework is clean and provides a solid foundation for future app-specific chains. The refactoring of the ZK-DEX program to use this new framework is a great showcase of its capabilities.
I've also noted several key optimizations, including the proof-free verification of empty batches and the early commitment of batches containing withdrawals, which will significantly improve the efficiency and user experience of the L2. The extensive additions to Docker and local testing infrastructure are also valuable contributions.
My review includes a few suggestions for improvement, mainly around code duplication and potential performance optimizations for future consideration. Overall, this is an excellent set of changes that greatly enhances the project's flexibility and performance.
Note: Security Review did not run due to the size of the PR.
…_balance_diffs get_balance_diffs() was updated to require a second parameter (native_token_scale_factor: Option<U256>) but this call site in app_execution.rs was not updated, causing compilation failure in all lint CI jobs.
…type_id - RLP decoder errors now include specific failure reasons (e.g., "insufficient data for length prefix", "length exceeds data bounds") instead of generic "invalid RLP encoding" messages. - Consolidate duplicated resolve_program_type_id() from deployer.rs and l1_committer.rs into ethrex_l2_common as the single source of truth for program type ID mapping.
Zena-park
merged commit d99df3e
into
feat/app-customized-framework
43 of 44 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Context
Stacked PR chain (
03→02→01→feat/app-customized-framework) had merge order issues.PR #6 (01→target) and PR #7 (02→01) were merged on 3/2, but PR #11 (03→02) was merged later.
This PR brings the remaining 19 commits into
feat/app-customized-framework.