Skip to content

feat(krakatoa): operate_exclusively toggles krakatoa features#1060

Open
mattac21 wants to merge 44 commits intomainfrom
ma/toggle
Open

feat(krakatoa): operate_exclusively toggles krakatoa features#1060
mattac21 wants to merge 44 commits intomainfrom
ma/toggle

Conversation

@mattac21
Copy link
Contributor

@mattac21 mattac21 commented Mar 9, 2026

Description

This PR adds a flag evm.mempool.operate-exclusively that will enable the KrakatoaMempool features. These features have been pulled out of the ExperimentalEVMMempool and into their own struct in krakatoa_mempool.go.

Mempool related system and integration tests now run two copies, they run once on the ExperimentalEVMMempool + CometBFT flood mempool, and once on the KrakatoaMempool+ CometBFT app mempool.

One slight modification was made to the ExperimentalEVMMempool though, when interacting with the legacypool, both mempools use the lifecycle hooks instead of directly modifying the legacypool. The ExperimentalEVMMempool calls the BroadcastTxFn when a tx is promoted via the onTxPromoted hook, instead of passing the BroadcastTxFn into the legacypool itself.

Closes: STACK-2091


Author Checklist

All items are required. Please add a note to the item if the item is not applicable and
please add links to any relevant follow up issues.

I have...

  • tackled an existing issue or discussed with a team member
  • left instructions on how to review the changes
  • targeted the main branch

@codecov
Copy link

codecov bot commented Mar 9, 2026

Codecov Report

❌ Patch coverage is 51.73210% with 209 lines in your changes missing coverage. Please review.
✅ Project coverage is 62.23%. Comparing base (bea1ab9) to head (16e15f1).

Files with missing lines Patch % Lines
mempool/krakatoa_mempool.go 51.37% 52 Missing and 89 partials ⚠️
mempool/mempool.go 54.41% 12 Missing and 19 partials ⚠️
mempool/txpool/legacypool/legacypool.go 63.15% 2 Missing and 12 partials ⚠️
rpc/backend/backend.go 0.00% 6 Missing ⚠️
server/start.go 0.00% 6 Missing ⚠️
server/server_app_options.go 20.00% 2 Missing and 2 partials ⚠️
rpc/backend/call_tx.go 0.00% 2 Missing ⚠️
rpc/backend/sign_tx.go 0.00% 2 Missing ⚠️
mempool/txpool/txpool.go 90.90% 0 Missing and 1 partial ⚠️
server/json_rpc.go 0.00% 1 Missing ⚠️
... and 1 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1060      +/-   ##
==========================================
- Coverage   64.11%   62.23%   -1.89%     
==========================================
  Files         331      331              
  Lines       23303    23840     +537     
==========================================
- Hits        14941    14837     -104     
- Misses       6946     7293     +347     
- Partials     1416     1710     +294     
Files with missing lines Coverage Δ
mempool/internal/queue/queue.go 61.03% <100.00%> (-29.76%) ⬇️
mempool/recheck_pool.go 63.90% <ø> (ø)
mempool/tx_tracker.go 55.38% <ø> (-6.16%) ⬇️
mempool/txpool/legacypool/tx_store.go 61.97% <100.00%> (-35.09%) ⬇️
server/config/config.go 41.61% <100.00%> (-5.29%) ⬇️
server/flags/flags.go 0.00% <ø> (ø)
mempool/txpool/txpool.go 50.70% <90.90%> (-2.24%) ⬇️
server/json_rpc.go 0.00% <0.00%> (ø)
x/vm/keeper/keeper.go 83.09% <0.00%> (-2.84%) ⬇️
rpc/backend/call_tx.go 47.84% <0.00%> (-13.90%) ⬇️
... and 7 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mattac21 mattac21 marked this pull request as ready for review March 11, 2026 03:57
@greptile-apps
Copy link

greptile-apps bot commented Mar 11, 2026

Greptile Summary

This PR introduces operate_exclusively as a runtime-configurable toggle (via evm.mempool.operate-exclusively in app.toml/flags) that determines which mempool implementation is used. When enabled, the new KrakatoaMempool (renamed from ExperimentalEVMMempool) is activated with its ABCI InsertTx/ReapTxs exclusive mode and cosmos tx recheck support. When disabled, the simplified ExperimentalEVMMempool runs in non-exclusive mode alongside CometBFT's clist-mempool, relying on CheckTx. The EVMMempool field on EVMD is widened to sdkmempool.ExtMempool and the VMKeeperI interface is decoupled via the new NotifiedMempool abstraction.

Key changes:

  • KrakatoaMempool extracted into its own file (krakatoa_mempool.go) with async EVM + cosmos insert queues, recheck, reap list, and tx tracker
  • ExperimentalEVMMempool significantly simplified: recheck, insert queues, txTracker, and reapList removed — now a lighter non-exclusive pool
  • EVMMempoolConfig.OperateExclusively, PendingTxProposalTimeout, and InsertQueueSize fields moved out of EVMMempoolConfig and into KrakatoaMempoolConfig
  • NotifiedMempool interface added to break concrete-type coupling between VMKeeperI and the mempool
  • Integration test suite extended with TestKrakatoaMempoolIntegrationTestSuite targeting the exclusive path

Confidence Score: 3/5

  • The PR is mostly safe structurally, but a logic issue with silently-dropped cosmos tx insertion errors in the exclusive path warrants attention before merging to production.
  • The architecture is well-structured and the feature toggle pattern is clean. However, in the KrakatoaMempool.InsertAsync path (used by NewInsertTxHandler), cosmos tx validation errors are silently dropped: the cosmosInsertQueue goroutine runs the ante handler asynchronously and any error is sent on a channel that InsertAsync never reads (it takes the default case and returns nil). This causes CometBFT to receive code=OK for invalid cosmos txs in exclusive mode, potentially leading to gossip of invalid transactions. Additionally, ExperimentalEVMMempool loses EVM tx recheck support entirely in this PR (the evmRechecker parameter is removed), which is a non-obvious behavioral regression for the non-exclusive path.
  • Pay close attention to mempool/krakatoa_mempool.go (InsertAsync cosmos error handling) and mempool/mempool.go (loss of recheck in ExperimentalEVMMempool).

Important Files Changed

Filename Overview
evmd/mempool.go Refactored to conditionally set up either KrakatoaMempool (exclusive) or ExperimentalEVMMempool (non-exclusive) based on the new operate-exclusively flag. CheckTxHandler/InsertTxHandler split is intentional per mode.
mempool/krakatoa_mempool.go New file implementing KrakatoaMempool. Contains copy-paste comment errors on onEVMTxPromoted/onEVMTxRemoved, a dead-code operateExclusively field (always true, never read by IsExclusive), and cosmos tx insertion errors are silently dropped in InsertAsync path.
mempool/mempool.go ExperimentalEVMMempool (non-exclusive mode) now simplified: removed recheck support (no evmRechecker/cosmosRechecker), insert queues, txTracker, and reapList. Contains misleading log field "error" in cosmos tx insertion path. Significant behavioral regression from old code.
evmd/tests/integration/create_app.go Adds exclusiveMempool param to CreateEvmd and NewAppOptionsWithFlagHomeAndChainID. Contains typo "exlcusiveMempool" in NewAppOptionsWithFlagHomeAndChainID parameter name.
server/server_app_options.go Adds GetShouldOperateExclusively helper reading from EVMMempoolOperateExclusively flag. Consistent with existing GetPendingTxProposalTimeout and GetMempoolInsertQueueSize patterns.
mempool/interface.go Introduces NotifiedMempool interface (HasEventBus + GetBlockchain) to decouple VMKeeperI from the concrete ExperimentalEVMMempool type. Clean abstraction for polymorphism.
evmd/app.go EVMMempool field type widened from *ExperimentalEVMMempool to sdkmempool.ExtMempool, enabling both KrakatoaMempool and ExperimentalEVMMempool to be stored.
x/vm/keeper/keeper.go evmMempool field type changed from *ExperimentalEVMMempool to NotifiedMempool interface, consistent with the new interface abstraction.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[configureEVMMempool] --> B{GetShouldOperateExclusively?}

    B -- true --> C[KrakatoaMempool]
    B -- false --> D[ExperimentalEVMMempool]

    C --> E[SetInsertTxHandler]
    C --> F[SetReapTxsHandler]
    C --> G[SetPrepareProposal\nNoCheckTxVerifier]
    C --> H[SetMempool]
    C --> I[NOT SetCheckTxHandler]

    D --> J[SetCheckTxHandler]
    D --> K[SetPrepareProposal\napp verifier]
    D --> L[SetMempool]
    D --> M[NOT SetInsertTxHandler/ReapTxs]

    C --> N[KrakatoaMempool.InsertAsync]
    N --> O{EVM tx?}
    O -- yes --> P[evmInsertQueue async]
    O -- no --> Q[cosmosInsertQueue async\n⚠️ errors silently dropped]

    C --> R[VMKeeperI.SetEvmMempool\nNotifiedMempool interface]
    D --> R
Loading

Comments Outside Diff (1)

  1. evmd/tests/integration/create_app.go, line 237 (link)

    Typo in parameter name

    exlcusiveMempool is misspelled — the l and c are transposed.

Last reviewed commit: a168921

@linear
Copy link

linear bot commented Mar 11, 2026

var ErrQueueFull = errors.New("queue full")

// New creates a new queue.
func New[Tx any](insert func(txs []*Tx) []error, maxSize int, logger log.Logger) *Queue[Tx] {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logger was unused

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the name of this mock was wrong (missing the I on the interface)

// The transactions can also be pre-filtered by the dynamic fee components to
// reduce allocations and load on downstream subsystems.
func (pool *LegacyPool) Pending(ctx context.Context, height *big.Int, filter txpool.PendingFilter) map[common.Address][]*txpool.LazyTransaction {
func (pool *LegacyPool) Rechecked(ctx context.Context, height *big.Int, filter txpool.PendingFilter) map[common.Address][]*txpool.LazyTransaction {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the ExperimentalEVMMempool calls Pending and the KrakataoMempool calls Rechecked.

// enough for the miner and other APIs to handle large batches of transactions;
// and supports pulling up the entire transaction when really needed.
type LazyTransaction struct {
Pool LazyResolver // Transaction resolver to pull the real transaction up
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused and made the filterAndWrapTxs helper have an extra param, so I removed it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved all krakatoa specific features into here. there is a bit of code duplication between this and the experimental evm mempool still. if we think it is worth breaking the common pieces out into more helper functions or a shared embedded struct, I can address in another pr

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of the mempool tests were specific to krakatoa, so moved them to krakatoa_mempool_test.go


// RemoveTx removes a tx from the tx tracker and does not record any metrics as
// it exits the tracker.
func (txt *txTracker) RemoveTx(hash common.Hash) {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unused

// handler for InsertTx, since the ABCI handler obfuscates the error's
// returned via codes, and we would like to have the full error to
// return to clients.
err := b.Mempool.Insert(b.Application.GetContextForCheckTx(txBytes), cosmosTx)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internally the mempool will use the context on the rechecker to run any validation that is happening during insert tx, so there is no need to specifically get the check context

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant