RPC node fails to re-sync after sequencer migration

[qzhodl]

Reproduction steps:

• Launch a full node on the new sequencer machine and sync it to the latest block.
• Stop the current sequencer components (geth, op-node, batcher, proposer, and challenger).
• On the new sequencer machine, stop the full node and restart it using the sequencer configuration.
• Because the boot node and static node IDs have changed, stop the RPC node, update its boot node and static node configuration, and restart it. This whole process may take more than 30 minutes.
• After the restart, the RPC node fails to continue syncing

[blockchaindevsh]

Looks like the migration is actually causing fork for my delta node:

# cast block 1046699 -r http://142.132.154.16:8545


baseFeePerGas        53
difficulty           0
extraData            0x00000000fa00000006
gasLimit             60000000
gasUsed              48649
hash                 0x205e90a9a1bed51d38989623e76e753eae1a2585f41be813e74787eb0fcfe427
logsBloom            0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
miner                0x4200000000000000000000000000000000000011
mixHash              0x1f578e11149c5e1d5204af41204082f16bce79a4dfc4dac6af74f756ef3fdb5e
nonce                0x0000000000000000
number               1046699
parentHash           0xc0d2b0a021c6c093cc3b7a303ab44d5c3b0c182afa491c35747ead9e2314c9b1
parentBeaconRoot     0xd0832c7b4a10cec2e368911b94d299c339c911c5f0dcf5b0ffb7a81bbd730b02
transactionsRoot     0xdf352d3fbc8d754a67ecc3dbcb70d849b849958ffb77a4bebe7fbe05129f01dc
receiptsRoot         0x8769a471dbb684bad3b584132b9fb1e5a196536b23ee782cf32386d32034a829
sha3Uncles           0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
size                 895
stateRoot            0x8ee1b3796f2fa592c0f79afb9fcde2ddc073c10b84f3ee4174af4c93eeab2db6
timestamp            1764601606 (Mon, 1 Dec 2025 15:06:46 +0000)
withdrawalsRoot      0x1fffe244007aa2058016d09960f0586a3a94a9fe9070552370c2271b37b2996f
totalDifficulty      
blobGasUsed          0
excessBlobGas        0
requestsHash         0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
transactions:        [
	
	blockHash            0x205e90a9a1bed51d38989623e76e753eae1a2585f41be813e74787eb0fcfe427
	blockNumber          1046699
	from                 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
	transactionIndex     0
	effectiveGasPrice    0
	
	hash                 0x1cdf3c1af1e8e2d15cbd60d5fbc726a3de31720b5c85e93fc4505eb52dc39693
	type                 126
	
	depositReceiptVersion 1
	gas                  1000000
	gasPrice             0
	input                0x098999be0000e5b30001bdb2000000000000000500000000692daefc000000000094b862000000000000000000000000000000000000000000000000000000000000000c0000000000000000000000000000000000000000000000000000000000000001c1407af8d04229c9de392a5babf65a20b283a7d11930c1c2f2ea52e70f2b9753000000000000000000000000385445d25164dff4038b0da6c9fa9548bbf9bd91000000000000000000000000
	mint                 0
	nonce                1046698
	r                    0
	s                    0
	sourceHash           48250592990511268591171981795332779522728006856309873362729683056413687225345
	to                   376793390874373408599387495934666716005045108757
	v                    0
	value                0
	
	                    
	            
]

The block hash of blockscout is 0x9acc51a01f3930a752dab2257139dbe9be0aa4a4f525d9dfd5d1c9c12eb95468 per here.

What's the block hash of sequencer for this height?

[blockchaindevsh]

Through bisection, I found my delta node disagrees with blockscout since block 1023489:

 # cast block 1023489 -r http://142.132.154.16:8545


baseFeePerGas        53
difficulty           0
extraData            0x00000000fa00000006
gasLimit             60000000
gasUsed              54225
hash                 0x562ba22a6f70eed00e71cbeca80c4374a95c7319fd760da163c869caae8d259b
logsBloom            0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
miner                0x4200000000000000000000000000000000000011
mixHash              0x7b98214cea5e281630560e5d34c1a4820d80e94027cf2f11a541410b6bbefa15
nonce                0x0000000000000000
number               1023489
parentHash           0xe7b83b147bafb18775d85abe47ba15d6097a37b6cd56ff41b0eda2feb9932e34
parentBeaconRoot     0xb680573f4afbf727e90817a176e1ec72009ba739cda3b406e6805ee28ccb7e06
transactionsRoot     0x2e0ba2fae71e8b50e52b7b7a1e7baa98861743440d5a655cf0e16a133e795c17
receiptsRoot         0xeaf6f5852e8be3264b7f20149a6f8cf42b8eb2e8a2d23a3fdccbc7795662f6ad
sha3Uncles           0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
size                 895
stateRoot            0x5f879f917c70aa3e795db60e79f6feb56b00ff9e58c218c7fe37926f14ca05dc
timestamp            1764555186 (Mon, 1 Dec 2025 02:13:06 +0000)
withdrawalsRoot      0x1fffe244007aa2058016d09960f0586a3a94a9fe9070552370c2271b37b2996f
totalDifficulty      
blobGasUsed          0
excessBlobGas        0
requestsHash         0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
transactions:        [
	
	blockHash            0x562ba22a6f70eed00e71cbeca80c4374a95c7319fd760da163c869caae8d259b
	blockNumber          1023489
	from                 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
	transactionIndex     0
	effectiveGasPrice    0
	
	hash                 0xb6a0dbf03a46fca1bf7a8ce2b75dd5be6a7c98bdfbe045cee44fef54f6a7263c
	type                 126
	
	depositReceiptVersion 1
	gas                  1000000
	gasPrice             0
	input                0x098999be0000e5b30001bdb2000000000000000000000000692cf970000000000094a94400000000000000000000000000000000000000000000000000000000000000090000000000000000000000000000000000000000000000000000000000000001738b922b5427620cc9ce000a879a5212c14bebe6e6351e77868976054131c1f3000000000000000000000000385445d25164dff4038b0da6c9fa9548bbf9bd91000000000000000000000000
	mint                 0
	nonce                1023488
	r                    0
	s                    0
	sourceHash           112796430069339640566074778101362578591595649278284497596170605086412232910077
	to                   376793390874373408599387495934666716005045108757
	v                    0
	value                0
	
	                    
	            
]

and there's such a log:

t=2025-12-02T02:18:53+0000 lvl=warn msg="L2 reorg: existing unsafe block does not match derived attributes from L1" err="random field does not match. expected: 0x7b98214cea5e281630560e5d34c1a4820d80e94027cf2f11a541410b6bbefa15. got: 0x4c1de9c839f703712855404dd782f604cbfb77e712fd1bbeddd77b349f062695" unsafe=0x4aeafc010711663b607e7161bcdb89d835b4edaf47bbee9c140d34021e074cb3:1023489 pending_safe=0xe7b83b147bafb18775d85abe47ba15d6097a37b6cd56ff41b0eda2feb9932e34:1023488

(The "L2 reorg" log only shows once in the op-node log.)

[blockchaindevsh]

By checking code, looks like there's such possibility:

1. The old sequencer published block N
2. The new sequencer(previously a full node) only got block N-1, then became the new sequencer, re-publishes a new block N
3. My node unfortunately got block N from the old sequencer, and the new block N from is silently dropped here.

[qzhodl]

Through bisection, I found my delta node disagrees with blockscout since block 1023489:

cast block 1023489 -r http://142.132.154.16:8545

baseFeePerGas 53
difficulty 0
extraData 0x00000000fa00000006
gasLimit 60000000
gasUsed 54225
hash 0x562ba22a6f70eed00e71cbeca80c4374a95c7319fd760da163c869caae8d259b
logsBloom 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
miner 0x4200000000000000000000000000000000000011
mixHash 0x7b98214cea5e281630560e5d34c1a4820d80e94027cf2f11a541410b6bbefa15
nonce 0x0000000000000000
number 1023489
parentHash 0xe7b83b147bafb18775d85abe47ba15d6097a37b6cd56ff41b0eda2feb9932e34
parentBeaconRoot 0xb680573f4afbf727e90817a176e1ec72009ba739cda3b406e6805ee28ccb7e06
transactionsRoot 0x2e0ba2fae71e8b50e52b7b7a1e7baa98861743440d5a655cf0e16a133e795c17
receiptsRoot 0xeaf6f5852e8be3264b7f20149a6f8cf42b8eb2e8a2d23a3fdccbc7795662f6ad
sha3Uncles 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347
size 895
stateRoot 0x5f879f917c70aa3e795db60e79f6feb56b00ff9e58c218c7fe37926f14ca05dc
timestamp 1764555186 (Mon, 1 Dec 2025 02:13:06 +0000)
withdrawalsRoot 0x1fffe244007aa2058016d09960f0586a3a94a9fe9070552370c2271b37b2996f
totalDifficulty
blobGasUsed 0
excessBlobGas 0
requestsHash 0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
transactions: [

blockHash 0x562ba22a6f70eed00e71cbeca80c4374a95c7319fd760da163c869caae8d259b
blockNumber 1023489
from 0xDeaDDEaDDeAdDeAdDEAdDEaddeAddEAdDEAd0001
transactionIndex 0
effectiveGasPrice 0

hash 0xb6a0dbf03a46fca1bf7a8ce2b75dd5be6a7c98bdfbe045cee44fef54f6a7263c
type 126

depositReceiptVersion 1
gas 1000000
gasPrice 0
input 0x098999be0000e5b30001bdb2000000000000000000000000692cf970000000000094a94400000000000000000000000000000000000000000000000000000000000000090000000000000000000000000000000000000000000000000000000000000001738b922b5427620cc9ce000a879a5212c14bebe6e6351e77868976054131c1f3000000000000000000000000385445d25164dff4038b0da6c9fa9548bbf9bd91000000000000000000000000
mint 0
nonce 1023488
r 0
s 0
sourceHash 112796430069339640566074778101362578591595649278284497596170605086412232910077
to 376793390874373408599387495934666716005045108757
v 0
value 0

]
and there's such a log:

t=2025-12-02T02:18:53+0000 lvl=warn msg="L2 reorg: existing unsafe block does not match derived attributes from L1" err="random field does not match. expected: 0x7b98214cea5e281630560e5d34c1a4820d80e94027cf2f11a541410b6bbefa15. got: 0x4c1de9c839f703712855404dd782f604cbfb77e712fd1bbeddd77b349f062695" unsafe=0x4aeafc010711663b607e7161bcdb89d835b4edaf47bbee9c140d34021e074cb3:1023489 pending_safe=0xe7b83b147bafb18775d85abe47ba15d6097a37b6cd56ff41b0eda2feb9932e34:1023488

(The "L2 reorg" log only shows once in the op-node log.)

Should the full node respect the batch from L1 and reorg its unsafe blocks?

[blockchaindevsh]

Should the full node respect the batch from L1 and reorg its unsafe blocks?

In fact, my node was of the correct block hash(0x4aeafc010711663b607e7161bcdb89d835b4edaf47bbee9c140d34021e074cb3) for 1023489 initially, but later reorged to the wrong block hash about 12 hours later during derivation:

t=2025-12-01T02:13:06+0000 lvl=info msg="Inserted new L2 unsafe block (synchronous)" hash=0x4aeafc010711663b607e7161bcdb89d835b4edaf47bbee9c140d34021e074cb3 number=1023489 newpayload_time=2.053ms fcu2_time=788.288µs total_time=2.842ms mgas=0.048649 mgasps=17.112756564714985


t=2025-12-02T02:18:53+0000 lvl=warn msg="L2 reorg: existing unsafe block does not match derived attributes from L1" err="random field does not match. expected: 0x7b98214cea5e281630560e5d34c1a4820d80e94027cf2f11a541410b6bbefa15. got: 0x4c1de9c839f703712855404dd782f604cbfb77e712fd1bbeddd77b349f062695" unsafe=0x4aeafc010711663b607e7161bcdb89d835b4edaf47bbee9c140d34021e074cb3:1023489 pending_safe=0xe7b83b147bafb18775d85abe47ba15d6097a37b6cd56ff41b0eda2feb9932e34:1023488
t=2025-12-02T02:18:53+0000 lvl=info msg="Inserted new L2 unsafe block" hash=0x562ba22a6f70eed00e71cbeca80c4374a95c7319fd760da163c869caae8d259b number=1023489 build_time=3.588ms insert_time=134.389ms total_time=137.977ms mgas=0.054225 mgasps=0.3929974372218574

The root cause is yet to be discovered.

[blockchaindevsh]

The reason is now clear: the new sequencer was previously using el-sync and has safe head drifts, when it becomes the new sequencer, the span batch will start from its own safe head+1, which will be dropped by other full nodes since it's not consecutive(this is added in holocene HF, src).

Conclusion

If we want to migrate a full node to the sequencer, recommend to:

1. op-geth using gcmode archive;
2. op-node use clsync; or elsync but safe/finalize is the same with the sequencer(the pending span batch is submitted to L1)