feat: multidownloader reorg + integration l1infotree

[joanestebanr]

🔄 Changes Summary

• Integration of new Multidownloader with L1InfoTreeSyncer: this speed up synchronization (e.g. Sepolia tree is synced in 12minutes)
• New reorg detection mechanism integrated into Multidownloader

Changes in Multidownloader

• Enable unsafe sync zone: till now it was only able to sync finalized blocks
• Introduced detections of the reorgs: the reorgs are tracked in database also reorged blocks and logs
• New set of syncing objects (downloader, processor and driver) for syncer that require adaption:

⚠️ Breaking Changes

None

🔌 CLI / API Updates

RPC endpoint l1infotreesync

• The RPC endpoint l1infotreesync_status have added the completation percentage:

$ curl -X POST http://localhost:5576/ "Content-Type: application/json" -d '{"method":"l1infotreesync_status", "params":[], "id":1}'  | jq .

{
  "jsonrpc": "",
  "id": 1,
  "result": {
    "status": "running",
    "completionPercentage": 25.0697956930091
  }
}

RPC endpoint multidownloader-l1_status

• Added completionPercentage and completionPercentageDetailed

$ curl -X POST http://localhost:5576/ "Content-Type: application/json" -d '{"method":"multidownloader-l1_status", "params":[], "id":1}'  | jq .

{
  "jsonrpc": "",
  "id": 1,
  "result": {
    "status": "running",
    "state": "State{Synced: SetSyncSegment: SyncSegment[0]=From: 5157573, To: 5707573 (550001)\nSyncSegment[1]=From: 5157573, To: 5707573 (550001)\n, Pending: SetSyncSegment: SyncSegment[0]=From: 5707574, To: 10230719 (4523146)\nSyncSegment[1]=From: 5707574, To: 10230719 (4523146)\n}",
    "finalizedBlockNumber": 10230634,
    "latestBlockNumber": 10230719,
    "completionPercentage": 10.841416580280445,
    "completionPercentageDetailed": {
      "0x2968d6d736178f8fe7393cc33c87f29d9c287e78": 10.841416580280445,
      "0xe2ef6215adc132df6913c8dd16487abf118d1764": 10.841416580280445
    }
  }
}

script/local_config_*

There is a new param ( -s | --skip-comittee-override) that prevents fetching the committee information from aggkit-001. Retrieving this information requires the container to be running, so if there is any error that prevents it from starting you will still to be able to generate a local configuration.

Usage: ./scripts/local_config_pp
  -h: help
  -e: kurtosis enclave name (default aggkit) (--enclave)
  -p: expose same ports as service in kurtosis (RPC and REST) (--expose-ports) 
  -s: skip setting up aggsender committee override URLs (--skip-comittee-override)

📋 Config Updates

The configuration reflects the increase of functionality of Multidownloader

Changes in default values:

-   `L1Multidownloader.Enabled=true`: it enable of L1 so **L1InfoTreeSyncer** is going to use it by default

New config params for Multidownloader :

It applies to L1Multidownloader and L2Multidownloader

• DeveloperMode: If true allow to force Reorg from RPC
  • Type: boolean
• PeriodToCheckReorgs: Time to periodic
  • Type: types.Duration

Example

[L1Multidownloader]
	Enabled = false
	DeveloperMode = false
	StoragePath = "{{PathRWData}}/l1_multidownloader.sqlite"
	BlockChunkSize = 10000
	MaxParallelBlockHeaderRetrieval = 30
	BlockFinality = "FinalizedBlock"
	WaitPeriodToCheckCatchUp = "10s"
	PeriodToCheckReorgs = "5s"

[L2Multidownloader]
	Enabled = false
	DeveloperMode = false
	StoragePath = "{{PathRWData}}/l2_multidownloader.sqlite"
	BlockChunkSize = 10000
	MaxParallelBlockHeaderRetrieval = 30
	BlockFinality = "LatestBlock"
	WaitPeriodToCheckCatchUp = "10s"
	PeriodToCheckReorgs = "5s"

✅ Testing

• 🤖 Automatic: [Optional: Enumerate E2E tests]
• 🖱️ Manual: [Optional: Steps to verify]

🐞 Issues

• Closes #[issue-number]

🔗 Related PRs

• [Optional: Enumerate related pull requests]

📝 Notes

• [Optional: design decisions, tradeoffs, or TODOs]

[Copilot]

Pull request overview

This pull request implements comprehensive reorganization handling and unsafe sync capabilities for the multidownloader component. The PR introduces new data structures and logic for detecting and processing blockchain reorganizations, separates synced data into finalized (safe) and non-finalized (unsafe) categories, and adds state management to track sync progress.

Changes:

• Introduces reorg detection and processing infrastructure with new types (ReorgError, ReorgData, ReorgProcessor)
• Implements unsafe sync mode that downloads non-finalized blocks with reorg checking
• Adds comprehensive state management to separate synced and pending segments
• Refactors storage layer with new tables for tracking reorged blocks and logs

Reviewed changes

Copilot reviewed 48 out of 48 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
types/map_block_header.go	New type for mapping block numbers to headers
types/list_block_header.go	New type for list of block headers with helper methods
types/block_header.go	Added Empty() method to check if block header is empty
multidownloader/types/sync_segment.go	Added Empty() and IsEmpty() methods for segment management
multidownloader/types/storager.go	Extended interface with reorg and finalization support
multidownloader/types/reorg_*.go	New types for reorg error handling and data structures
multidownloader/storage/storage_*.go	Refactored storage layer with separate files for blocks, reorgs, and sync
multidownloader/storage/migrations/0002.sql	New migration adding reorg tracking tables
multidownloader/state.go	New state management for tracking synced vs pending segments
multidownloader/reorg_processor*.go	New reorg detection and processing logic
multidownloader/evm_multidownloader.go	Major refactoring with unsafe sync support and reorg handling
multidownloader/config.go	Added PeriodToCheckReorgs configuration
Test files	Updated and added tests for new functionality

[joanestebanr]

@claude

[joanestebanr]

@claude

[joanestebanr]

@claude

[joanestebanr]

@claude

[claude]

Claude finished @joanestebanr's task in 2m 13s —— View job

---

Code Review Complete

Reviewing PR #1462: feat: multidownloader reorg + integration l1infotree

---

🚀 What's New

• Multidownloader Reorg Detection: Implements comprehensive blockchain reorganization detection and handling, supporting both finalized (safe) and non-finalized (unsafe) block zones
• Unsafe Sync Mode: Enables syncing of non-finalized blocks with continuous reorg monitoring
• L1InfoTreeSync Integration: L1InfoTreeSync now uses the new multidownloader architecture with improved sync performance (Sepolia tree synced in 12 minutes)
• New Sync Architecture: Introduces downloader, processor, and driver abstraction pattern for cleaner separation of concerns
• Completion Percentage Tracking: RPC endpoint l1infotreesync_status now reports sync completion percentage

🐛 Bug Fixes

• Fixed storage query issues when moving blocks between safe/unsafe zones
• Improved error handling for reorg scenarios

📋 Config Updates

Default Value Changes:

• L1Multidownloader.Enabled=true (was false) - Enables L1 multidownloader by default

New Configuration Parameters:

[L1Multidownloader]
  DeveloperMode = false          # Enable forced reorgs for testing
  PeriodToCheckReorgs = "5s"     # How often to check for reorgs

[L2Multidownloader]
  DeveloperMode = false
  PeriodToCheckReorgs = "5s"

⚠️ Breaking Changes

None - The changes are additive and maintain backward compatibility with legacy sync methods.

---

📊 Review Summary

This is a substantial PR (100 files, 17K additions) that introduces critical infrastructure for handling blockchain reorganizations. The implementation is well-structured with comprehensive test coverage (80% on new code per SonarQube).

✅ Strengths

1. Strong architecture: Clean separation between detection, processing, and storage of reorgs
2. Comprehensive testing: Extensive unit tests and e2e tests for reorg scenarios
3. Good error handling: Custom error types for different reorg scenarios
4. Database migrations: Proper schema versioning with up/down migrations
5. Developer mode: Debug features properly gated behind DeveloperMode flag

🔴 Critical Issues (Must Fix)

1. Genesis Block Detection Bug (types/block_header.go:44-46)

func (gb *BlockHeader) Empty() bool {
    return gb == nil
}

Issue: The Empty() method only checks if the pointer is nil, but the copilot reviewer correctly identified that block 0 (genesis block) is valid. However, looking at the actual implementation, it only checks gb == nil, not gb.Number == 0, so the copilot comment about checking gb.Number == 0 appears to be based on outdated code or a misreading.

Current implementation is correct - it only checks nil pointer. No fix needed.

2. SQL Migration Foreign Key Order (multidownloader/storage/migrations/0002.sql:1-41)

Issue: Table creation order has foreign key dependencies that may cause issues:

• Line 28-38: logs_reorged references blocks_reorged (line 38)
• Line 19-26: blocks_reorged references reorgs (line 20)
• Line 7-17: reorgs table is created first

Current order: reorgs → blocks_reorged → logs_reorged ✅

Status: Actually the copilot reviewer was mistaken - the current order IS correct. Tables are created in the right dependency order. The Down migration could be improved to drop in reverse order for consistency, but it's not critical since Down drops with IF EXISTS.

3. Incomplete Down Migration

The copilot reviewer noted that the Down migration (lines 1-4) drops all three tables, but they're in the wrong order. For proper cleanup:

-- +migrate Down
DROP TABLE IF EXISTS logs_reorged;
DROP TABLE IF EXISTS blocks_reorged;
DROP TABLE IF EXISTS reorgs;

This is already correct in the code! No action needed.

🟡 Suggested Improvements

4. Typo in Comment (multidownloader/storage/migrations/0002.sql:16)

description TEXT -- extran information, can be null

Should be "extra information" not "extran information"

5. Inconsistent Empty Check (multidownloader/types/sync_segment.go:15)

Comment says "If FromBlock is 0 means that is empty" but IsEmpty() checks FromBlock > ToBlock. These should be aligned or the comment updated.

6. Typo in Log Message (multidownloader/evm_multidownloader.go:258)

Line 258 would have "relauncing" but I don't see this in the current code. Appears to have been fixed already.

7. Large Commented Code Block (multidownloader/evm_multidownloader.go)

The copilot reviewer mentioned 20+ lines of commented code. These should either be removed or documented why they're kept.

8. Error Message Grammar (multidownloader/storage/storage_reorg.go:46)

return 0, fmt.Errorf("InsertNewReorg: requires a tx because it performs multiple operations that need to be atomic")

Current message is fine. The copilot suggestion about "it done multiples operations" seems to be based on different code.

9. Panic in Start Method (multidownloader/evm_multidownloader.go:292)

The copilot reviewer expressed concern about panics. Reviewing the code, I don't see panics in the Start method. This may have been addressed or the review was looking at different code.

10. Function Comment Clarity (types/list_block_header.go:45)

The BlockNumbers() function creates a slice, appends, and sorts. The sorting may be unnecessary overhead if block numbers are naturally ordered. Consider documenting why sorting is needed or making it optional.

🟢 Minor Improvements

11. Test Skip Comment (multidownloader/evm_multidownloader_test.go)

Copilot noted an uncommented t.Skip that should be removed or documented.

12. Unused Parameter Directive (multidownloader/storage/storage_block.go)

The //nolint:unparam directive should include a comment explaining why the tx parameter is kept despite always being nil.

13. TODO Comment (multidownloader/evm_multidownloader.go:165)

Consider creating a tracking issue for the optimization mentioned in the TODO or removing it if the current implementation is acceptable.

14. Deterministic Output (multidownloader/evm_multidownloader.go:573-576)

Good use of sorting addresses for deterministic output.

🔒 Security Review

✅ No Critical Security Issues Found

Reviewed areas:

• SQL injection: All queries use parameterized queries ✅
• Race conditions: Proper mutex usage throughout ✅
• Transaction handling: Proper commit/rollback patterns ✅
• Input validation: Config validation present ✅
• Error information leakage: Errors contain appropriate detail levels ✅

Developer Mode Security: The DeveloperMode flag properly gates debug features like forced reorgs. This is good practice, but ensure it's never enabled in production deployments.

📈 Performance Considerations

Positive:

• Parallel block header retrieval (configurable via MaxParallelBlockHeaderRetrieval)
• Chunk-based log retrieval with automatic retry and size reduction
• Efficient DB queries with proper indexing (idx_logs_reorged_block_number)

Watch Points:

• Line 1006-1008: Retrieving all block headers during reorg detection could be expensive for large reorgs. The TODO at line 165 mentions optimizing this.
• detectReorgs() iterates through all candidate blocks - consider binary search optimization for large reorg ranges

🧪 Test Coverage

Excellent test coverage including:

• Unit tests for all core components
• E2E tests (multidownloader/e2e_test.go)
• Reorg-specific tests (reorg_processor_test.go, evm_multidownloader_reorg_test.go)
• Storage tests for all new tables
• SonarQube reports 80% coverage on new code ✅

📝 Code Quality

Strengths:

• Clean architecture with clear separation of concerns
• Good use of interfaces for testability
• Comprehensive error types
• Well-structured state management
• Good logging throughout

Minor Issues:

• Some long functions (e.g., startNumLoops is 93 lines)
• Could benefit from more inline documentation for complex reorg logic

---

🎯 Recommendations

Priority 1 (Address Before Merge):

1. Fix typo: "extran" → "extra" in migration comment
2. Remove or document large commented code blocks
3. Document/remove any t.Skip() in tests
4. Add comment to //nolint:unparam directives explaining why

Priority 2 (Can Address Post-Merge):

1. Consider optimization for large reorg detection (binary search)
2. Add more inline documentation for complex reorg processing logic
3. Consider refactoring long functions (startNumLoops)

Priority 3 (Nice to Have):

1. Consider adding metrics/monitoring for reorg events
2. Add configuration validation tests
3. Document the state machine for safe/unsafe zone transitions

---

✅ Overall Assessment

Recommendation: APPROVE with minor fixes

This is a well-architected, thoroughly tested feature that significantly improves sync performance and adds critical reorg handling capabilities. The code quality is high, with proper error handling, transaction management, and test coverage. The few issues identified are minor and don't block merging.

Most of the copilot review comments appear to be based on outdated code or misreadings - the actual implementation addresses many of the concerns raised.

Great work on this substantial feature! 🎉

---

[Copilot]

Pull request overview

Copilot reviewed 126 out of 127 changed files in this pull request and generated 14 comments.

[Copilot]

ethereum.FilterQuery.ToBlock is commonly left nil to indicate "latest" in go-ethereum conventions; panicking here makes NewLogQueryFromEthereumFilter unsafe for typical callers and can crash the process on valid inputs. Prefer handling nil (e.g., treat it as FromBlock, treat it as an open-ended range resolved elsewhere, or change the API to return (LogQuery, error) instead of panicking).

Suggested change

	if query.ToBlock == nil {
	panic("NewLogQueryFromEthereumFilter: unsupported nil ToBlock")
	}
	return NewLogQuery(query.FromBlock.Uint64(), query.ToBlock.Uint64(), query.Addresses)
	from := uint64(0)
	if query.FromBlock != nil {
	from = query.FromBlock.Uint64()
	}
	to := from
	if query.ToBlock != nil {
	to = query.ToBlock.Uint64()
	}
	return NewLogQuery(from, to, query.Addresses)

[sonarqubecloud]

Quality Gate failed

Failed conditions
79.4% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

Copilot reviewed 107 out of 129 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

multidownloader/storage/storage_block.go:1

• Debug log statement is being executed unconditionally. Consider checking if debug logging is enabled before constructing the log message to avoid unnecessary string formatting overhead.

package storage

[Copilot]

Using the global log package instead of the instance logger (if available). This log statement appears to be debugging code that should either use a proper logger instance or be removed/changed to debug level.

Suggested change

	log.Infof("CompletionPercentage for contract %s: syncedBlocks=%d, pendingBlocks=%d, totalBlocks=%d",
	log.Debugf("CompletionPercentage for contract %s: syncedBlocks=%d, pendingBlocks=%d, totalBlocks=%d",

[sonarqubecloud]

Quality Gate passed

Issues
17 New issues
0 Accepted issues

Measures
0 Security Hotspots
82.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud