execution: restore commitment deferred updates hook mechanism#18917
Open
Giulio2002 wants to merge 118 commits intomainfrom
Open
execution: restore commitment deferred updates hook mechanism#18917Giulio2002 wants to merge 118 commits intomainfrom
Giulio2002 wants to merge 118 commits intomainfrom
Conversation
- Re-implement ApplyDeferredUpdates as sequential instead of parallel - Add DeferredHooker interface for deferring branch updates to flush hooks - Add FlushHooks/AddFlushHook to SharedDomains for deferred execution - Enable deferred hooker in fork validation mode at chain tip - Call FlushHooks in MergeExtendingFork to apply deferred updates - Capture commitment writes during FlushHooks and add to past changesets - This ensures changeset3 table includes commitment data for unwinding Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Giulio2002
requested review from
AskAlexSharov,
mh0lt,
sudeepdino008 and
yperbasis
as code owners
Giulio2002
changed the title
you can now start claude on your erigon github repo dir, then ask "change step size of datadir /Users/wmitsuda/eth-nodes/erigon34-mainnet-test to 781250"
Giulio2002
changed the title
#18865) Why do we include `MainnetBootstrapNodes` for Sepolia/Gnosis/Chiado?
…dundant allocations (#18850) Removes unnecessary slice allocation and append operations in `KzgCommitmentToVersionedHash`.
…/transactions > 16MB (#18932) fixes #18920 fixes #15881 Another fun day in the office. This issue surfaced in gas-benchmarks at high mgas scenarios in the form of incorrect ReceiptsRoot and/or TransactionsRoot hash for correct receipts/transactions, e.g.: ``` backend.go:667: WARN[02-02|16:38:50.484] Could not validate block err="[5/5 Execution] invalid block, receiptHash mismatch: 36989b292c72b2b97fc4787453573716bac13679ad0c47ca04583bf2a2b5190b != 8133caae197e6049123faab206e79972cd392bc749d403321280fc0c0e726763, headerNum=6, 1bf17d04f638c651093047251c75b404337a24eeb6364c95ec257e56685aaee6" ``` Our code assumes that we will never receive a struct for RLP encoding with more than 16_777_216 bytes Alas this did happen for large receipts that can be generated at high mgas and was flagged in Kamil's gas-benchmarking tests. Example logs (can see the culprits being called with 17400275 byte length lists): ``` --- debug --- generateRlpPrefixLen(17400275) --- debug --- GenerateStructLen(17400282) --- debug --- generateByteArrayLen(17400275, 0) ``` The incorrect RLP prefix len then lead to incorrect trie roots. A test reproducing this has been added and fixed.
This unit test reproduces a panic I've encountered when requesting trie
witness for an account and storage slot of another account in one batch
(i.e. in one call to `GenerateWitness`.
`MultiKeyWitness_AccountWithSingletonStorage` fails with:
```
panic: runtime error: index out of range [16] with length 16 [recovered]
panic: runtime error: index out of range [16] with length 16
```
This happens during the unfold of the storage key, and I think the
reason is that the fold didn't happen correctly before that.
The full trace is :
```
go test -test.fullpath=true -timeout 600s -run ^Test_WitnessTrie_GenerateWitness$/^MultiKeyWitness_AccountWithSingletonStorage$ github.com/erigontech/erigon/execution/commitment
addr (prefix 0x5) 709b0758083f61d375bc02b41df4f91929e18fda
addr (prefix 0xa) 6b6fed27ff8974bac0cafd9ad05692b13619e738
storage 964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb -> 964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb
(1/3) plainKey [709b0758083f61d375bc02b41df4f91929e18fda] Flags: [+Balance], Balance: [100] hashedKey [050804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07] currentKey []
needUnfolding root, rootChecked = false
unfold 1: activeRows: 0
unfold root: touched: false present: false {loaded=false}
unfoldBranchNode prefix '00', nibbles [] depth 1 row 0 ''
needUnfolding root, rootChecked = true
set downHasheKey=[050804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07]
updateCell 709b0758083f61d375bc02b41df4f91929e18fda => Flags: [+Balance], Balance: [100]
(2/3) plainKey [6b6fed27ff8974bac0cafd9ad05692b13619e738] Flags: [+Balance], Balance: [200] hashedKey [0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f02040308] currentKey []
needUnfolding root, rootChecked = true
cpl=0 cell.hashedExtension=[050804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07] hashedKey[depth=0:]=[0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f02040308]
unfold 1: activeRows: 0
unfold root: touched: true present: true {loaded=Account addr=709b0758083f61d375bc02b41df4f91929e18fda balance=100 nonce=0 codeHash=EMPTY hashedExtension=050804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07}
unfolded cell (0, 5, depth=1) {loaded=Account addr=709b0758083f61d375bc02b41df4f91929e18fda balance=100 nonce=0 codeHash=EMPTY hashedExtension=0804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07}
currentKey [] needUnfolding cell (0, a, depth=1) cell.hash=[]
updateCell setting (0, a, depth=1)
set downHasheKey=[0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f02040308]
updateCell 6b6fed27ff8974bac0cafd9ad05692b13619e738 => Flags: [+Balance], Balance: [200]
(3/3) plainKey [6b6fed27ff8974bac0cafd9ad05692b13619e738964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb] Flags: [+Storage], Storage: [964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb] hashedKey [0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f0204030803050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d0305] currentKey []
currentKey [] needUnfolding cell (0, a, depth=1) cell.hash=[]
cpl=62 cell.hashedExtension=[0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f02040308] hashedKey[depth=1:]=[0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f0204030803050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d0305]
unfold 63: activeRows: 1
upCell (0, a, updepth=1) touched: true present: true
unfolded cell (1, 8, depth=64) {loaded=Account addr=6b6fed27ff8974bac0cafd9ad05692b13619e738 balance=200 nonce=0 codeHash=EMPTY}
currentKey [0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f020403] needUnfolding cell (1, 8, depth=64) cell.hash=[]
updateCell setting (1, 8, depth=64)
set downHasheKey=[03050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d0305]
updateCell 6b6fed27ff8974bac0cafd9ad05692b13619e738964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb => Flags: [+Storage], Storage: [964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb]
fold [0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f020403] activeRows: 2 touchMap: 0000000100000000 afterMap: 0000000100000000
fold: parent (0, a, depth=1)
fold: (row=1, {8}, depth=64) prefix [0a0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f020403] touchMap: 0000000100000000 afterMap: 0000000100000000
formed leaf (1 8, depth=64) [1a196bb56b96a66a835b2d1563c2d3cf630905829dd8f0778a89b261ede7f243] {loaded=Account Storage addr=6b6fed27ff8974bac0cafd9ad05692b13619e738 balance=200 nonce=0 codeHash=EMPTY addr[s]=6b6fed27ff8974bac0cafd9ad05692b13619e738964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb storage=964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb hashedExtension=03050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d0305}
fold [] activeRows: 1 touchMap: 0000010000100000 afterMap: 0000010000100000
fold: parent is root {loaded=Account addr=709b0758083f61d375bc02b41df4f91929e18fda balance=100 nonce=0 codeHash=EMPTY hashedExtension=050804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c07}
fold: (row=0, {5,A}, depth=1) prefix [] touchMap: 0000010000100000 afterMap: 0000010000100000
accountLeafHashWithKey {a0a7a0052a6938344bebbcb671fc857203d28cd97390aa38d826e7bc2c9aed4d9f} (memorised) for [0804010901090c090308080c0d0f07080e0e00010507070c03040a0d0c050609010d00080008090f0701040501060f0701010a0a02000c01000b0300000c0710]=>[f8448064a056e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470]
leafHashWithKeyVal(singleton=true) {a0d3385607f70e0ad68d89bd41c7acb5f5cfc91ad6cf158b4a792fbaf412ac4bb2} for [03050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d030510]=>[964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb] (hashedExtension(len=63)=03050d060e08030509010b060708010d0a0b060309060d010d00070305080703000a06070e000a0b0a0c080e0305040103030e060f0d08080d0f010c000d03, accountAddr=6b6fed27ff8974bac0cafd9ad05692b13619e738, storageAddr=6b6fed27ff8974bac0cafd9ad05692b13619e738964dfdc79e8d534373661cfd66d74fec1e1b89491ab7236e4b75216290cf2beb, )
accountLeafHashWithKey {a09ecd89e2b87e3a640e3ef846b204cb2afb0c93912702824b2f06322691c34df6} (memorised) for [0109060b0b05060b09060a06060a0803050b020d010506030c020d030c0f0603000900050802090d0d080f000707080a08090b0206010e0d0e070f0204030810]=>[f8458081c8a0d3385607f70e0ad68d89bd41c7acb5f5cfc91ad6cf158b4a792fbaf412ac4bb2a0c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470]
} [3a242aefe2cea9cfb59d87890c905bbd31ef058f95ac671c214ae9dd4d739c93]
root hash 3a242aefe2cea9cfb59d87890c905bbd31ef058f95ac671c214ae9dd4d739c93 updates 3
--- FAIL: Test_WitnessTrie_GenerateWitness (0.00s)
--- FAIL: Test_WitnessTrie_GenerateWitness/MultiKeyWitness_AccountWithSingletonStorage (0.00s)
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed_test.go:2442: MultiKeyWitness_AccountWithSingletonStorage
panic: runtime error: index out of range [16] with length 16 [recovered]
panic: runtime error: index out of range [16] with length 16
goroutine 36 [running]:
testing.tRunner.func1.2({0x100fcc7a0, 0x1400022a1b0})
/opt/homebrew/Cellar/go/1.24.2/libexec/src/testing/testing.go:1734 +0x1ac
testing.tRunner.func1()
/opt/homebrew/Cellar/go/1.24.2/libexec/src/testing/testing.go:1737 +0x334
panic({0x100fcc7a0?, 0x1400022a1b0?})
/opt/homebrew/Cellar/go/1.24.2/libexec/src/runtime/panic.go:792 +0x124
github.com/erigontech/erigon/execution/commitment.(*HexPatriciaHashed).unfold(0x14000300000, {0x14000290080?, 0x80?, 0x80?}, 0x1)
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed.go:1812 +0x738
github.com/erigontech/erigon/execution/commitment.(*HexPatriciaHashed).GenerateWitness.func1({0x14000290080, 0x80, 0x80}, {0x140002501c0, 0x34, 0x34}, 0x0?)
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed.go:2479 +0x740
github.com/erigontech/erigon/execution/commitment.(*Updates).HashSort(0x1400013e240, {0x1010201d8, 0x1014bd0a0}, 0x0, 0x140000d4060)
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/commitment.go:1724 +0x314
github.com/erigontech/erigon/execution/commitment.(*HexPatriciaHashed).GenerateWitness(0x14000300000, {0x1010201d8, 0x1014bd0a0}, 0x1400013e240, 0x0, {0x0, 0x0})
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed.go:2436 +0x25c
github.com/erigontech/erigon/execution/commitment.Test_WitnessTrie_GenerateWitness.func1(0x14000256540, 0x14000149668, {0x1400003bf28, 0x2, 0x14000149628?}, {0x1400003b57e, 0x2, 0x0?})
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed_test.go:2038 +0x3b8
github.com/erigontech/erigon/execution/commitment.Test_WitnessTrie_GenerateWitness.func16(0x14000256540)
/Users/user/Documents/repos/erigontech/erigon-support/execution/commitment/hex_patricia_hashed_test.go:2473 +0x97c
testing.tRunner(0x14000256540, 0x14000209c40)
/opt/homebrew/Cellar/go/1.24.2/libexec/src/testing/testing.go:1792 +0xe4
created by testing.(*T).Run in goroutine 35
/opt/homebrew/Cellar/go/1.24.2/libexec/src/testing/testing.go:1851 +0x374
FAIL github.com/erigontech/erigon/execution/commitment 0.284s
FAIL
```
---------
Co-authored-by: antonis19 <antonis19@users.noreply.github.com>
Contributor
Author
|
ah I see now. well we have a lot of tests covering it. just not all of them. e.g spectest should go through this logic + does hive tests and kurtosis |
## Summary - Add a new reference skill (`.claude/skills/erigon-datadir/SKILL.md`) for manipulating Erigon datadirs - Covers safe datadir duplication with precondition checks, APFS copy-on-write support, and disk space validation - Non-user-invocable; loaded automatically when Claude detects datadir operations ## Test plan - [x] Tested duplication of an 820GB mainnet datadir to a temporary directory - [x] All precondition checks passed - [x] APFS CoW clone completed successfully, chaindata/ deleted from destination - [x] Confirmed zero extra disk space consumed (CoW) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Giulio2002
force-pushed
the
apply-commitment-updates-hook
branch
from
5611dea to
af52b93
Compare
Giulio2002
requested review from
bloxster,
canepat,
domiwei and
taratorio
as code owners
…igontech/erigon into apply-commitment-updates-hook
Giulio2002
force-pushed
the
apply-commitment-updates-hook
branch
from
05a3ab6 to
1654cb0
Compare
Giulio2002
force-pushed
the
apply-commitment-updates-hook
branch
from
d5ec877 to
755a8b7
Compare
Giulio2002
force-pushed
the
apply-commitment-updates-hook
branch
from
5364c15 to
26844fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Impact: