Nodes stuck after v0.8.0 upgrade - info tree cannot be verified

[SimonKraus]

Multiple node operators in Wormholes guardian network experienced the same issue.
The node was running fine after the update but roughly 10 minutes later they stopped working and are looping the same errors ever since:

[INFO] [08-06|08:23:25.317] [2/15 L1InfoTree] Starting L1 Info Tree stage
[INFO] [08-06|08:23:25.317] Starting L1 syncer thread
[INFO] [08-06|08:23:25.380] [2/15 L1InfoTree] Checking for L1 info tree updates, logs count:32
[EROR] [08-06|08:23:25.558] [2/15 L1InfoTree] Could not find an l1 info tree update by root root=0xdb6bce55def1747744c63df7141da2a96f415b881410b2e6c521a8e885b947fa
[INFO] [08-06|08:23:26.371] Stopping L1 syncer thread
[INFO] [08-06|08:23:26.559] [2/15 L1InfoTree] Finished L1 Info Tree stage
[EROR] [08-06|08:23:26.559] Error while executing stage              err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"
[EROR] [08-06|08:23:26.559] Staged Sync                              err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"

[drcliche]

Any update to this? Seeing same issue when trying to upgrade, as instructed by deadline of tomorrow.

[Inbell1s]

Can confirm we encountered this issue during the last update and now once again:

Aug 16 20:57:37 ML-EU-XLAYER-1 cdk-erigon[906281]: [INFO] [08-16|20:57:37.653] [2/15 L1InfoTree] Starting L1 Info Tree stage
Aug 16 20:57:37 ML-EU-XLAYER-1 cdk-erigon[906281]: [INFO] [08-16|20:57:37.653] Starting L1 syncer thread
Aug 16 20:57:37 ML-EU-XLAYER-1 cdk-erigon[906281]: [INFO] [08-16|20:57:37.940] [2/15 L1InfoTree] Checking for L1 info tree updates, logs count:148
Aug 16 20:57:38 ML-EU-XLAYER-1 cdk-erigon[906281]: [EROR] [08-16|20:57:38.603] [2/15 L1InfoTree] Could not find an l1 info tree update by root root=0xa499b2cc0258742a88a1854cd9f2166abf93533e4c9b14b87c77fbede70226e2
Aug 16 20:57:38 ML-EU-XLAYER-1 cdk-erigon[906281]: [INFO] [08-16|20:57:38.930] Stopping L1 syncer thread
Aug 16 20:57:39 ML-EU-XLAYER-1 cdk-erigon[906281]: [INFO] [08-16|20:57:39.604] [2/15 L1InfoTree] Finished L1 Info Tree stage
Aug 16 20:57:39 ML-EU-XLAYER-1 cdk-erigon[906281]: [EROR] [08-16|20:57:39.604] Error while executing stage              err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"
Aug 16 20:57:39 ML-EU-XLAYER-1 cdk-erigon[906281]: [EROR] [08-16|20:57:39.604] Staged Sync                              err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"

[jackeycui7]

i use v0.8.1, seeing the same issue

[napallday]

having the same issue with v0.8.1

[tomblible]

[EROR] [08-18|14:34:25.655] Staged Sync err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"
[INFO] [08-18|14:34:26.155] [2/15 L1InfoTree] Starting L1 Info Tree stage
[INFO] [08-18|14:34:26.155] Starting L1 syncer thread
[INFO] [08-18|14:34:26.494] [2/15 L1InfoTree] Checking for L1 info tree updates, logs count:122
[EROR] [08-18|14:34:27.930] [2/15 L1InfoTree] Could not find an l1 info tree update by root root=0xf877f20adad1eb3f12abe7dca34e1758e346c1b0929b2fe571f160e323b84c60
[INFO] [08-18|14:34:28.613] Stopping L1 syncer thread
[INFO] [08-18|14:34:28.931] [2/15 L1InfoTree] Finished L1 Info Tree stage
[EROR] [08-18|14:34:28.931] Error while executing stage err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"
[EROR] [08-18|14:34:28.931] Staged Sync err="[2/15 L1InfoTree] CheckForInfoTreeUpdates: RollbackL1InfoTree: first fork12 info tree update v2 transaction shows our info tree cannot be verified"
[INFO] [08-18|14:34:29.431] [2/15 L1InfoTree] Starting L1 Info Tree stage
[INFO] [08-18|14:34:29.431] Starting L1 syncer thread
[INFO] [08-18|14:34:29.792] [2/15 L1InfoTree] Checking for L1 info tree updates, logs count:122

[tomblible]

Why aren't the code bugs maintained? Many nodes can't synchronize due to various bugs？？？

[SimonKraus]

You can try resyncing from scratch but to prevent OutOfMemory issues add these to the config:

zkevm.increment-tree-always: false
zkevm.rebuild-tree-after: 10000000

[jackeycui7]

Thanks, brother. Do you mean deleting the chaindata folder and resyncing, or adding

zkevm.increment-tree-always: false
zkevm.rebuild-tree-after: 10000000 

to the config and restarting?

[SimonKraus]

@jackeycui7 sadly there's no way to recover from the above mentioned issue.
Yes , wiping the chaindata, then adding the parameters to the config and resyncing from genesis.

It should be quite quick, maybe half a day to sync up again.

[jackeycui7]

When I restarted, I still encountered this problem. I found that this problem always appeared in the [2/15 L1InfoTree] Process log, and always appeared around the progress of 23000. It seems that there is a problem with the log at a certain height. Do you also encounter this problem here?

[jackeycui7]

@SimonKraus thank you bro,i will try

[ericjohncarlson]

We're stuck with

[EROR] [08-19|23:36:35.111] Staged Sync                              err="highestVerification.BatchNo > 0, trace: [stageloop.go:105 panic.go:792 stage_l1_syncer.go:212 stages.go:271 sync.go:555 sync.go:425 stageloop.go:154 stageloop.go:73 asm_amd64.s:1700]"

[napallday]

We're still encountering OOM issues despite using the following configuration:

zkevm.increment-tree-always: false
zkevm.rebuild-tree-after: 10000000

@SimonKraus, could you share the full contents of your config.yaml? That might help us troubleshoot this further. Thanks!

[SimonKraus]

@napallday this might be a messy one cause we trial&&error'ed a lot but this is what got us back into the game (until all nodes died again tonight with other errors):

datadir: /data/.xlayer/data
chain: xlayer-mainnet
db.size.limit: 8TB
http: true
private.api.addr: localhost:9091
zkevm.l2-chain-id: 196
zkevm.l2-sequencer-rpc-url: https://rpc.xlayer.tech
zkevm.l2-datastreamer-url: stream.xlayer.tech:8800
zkevm.l1-chain-id: 1
zkevm.l1-rpc-url: http://OUR.LOCAL.L1:8545

zkevm.address-sequencer: "0xAF9d27ffe4d51eD54AC8eEc78f2785D7E11E5ab1"
zkevm.address-zkevm: "0x2B0ee28D4D51bC9aDde5E58E295873F61F4a0507"
zkevm.address-admin: "0x491619874b866c3cDB7C8553877da223525ead01"
zkevm.address-rollup: "0x5132A183E9F3CB7C848b0AAC5Ae0c4f0491B7aB2"
zkevm.address-ger-manager: "0x580bda1e7A0CFAe92Fa7F6c20A3794F169CE3CFb"

zkevm.l1-rollup-id: 3
zkevm.l1-first-block: 19218658
zkevm.l1-block-range: 2000
zkevm.l1-query-delay: 1000
zkevm.rpc-ratelimit: 500
zkevm.increment-tree-always: false
zkevm.rebuild-tree-after: 10000000

externalcl: true
http.api: [eth, debug, net, trace, web3, erigon, zkevm]
ws.api: [eth, debug, net, trace, web3, erigon, zkevm]
http.addr: 0.0.0.0
ws.addr: 0.0.0.0
http.vhosts: any
http.corsdomain: any
ws.port: 8546
http.port: 8545
metrics: true
ws: true

batchSize: 512M
etl.bufferSize: 512MB

[napallday]

@SimonKraus Cool bro! Could I also know how much memory do your instance have? I tried with 72GB memory, but it still got OOM...😰