feat(gossip): add partial messages extension for PeerDAS#8748
feat(gossip): add partial messages extension for PeerDAS#8748raulk wants to merge 2 commits intoChainSafe:unstablefrom
Conversation
Summary of ChangesHello @raulk, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly upgrades the network's data propagation capabilities by introducing partial message support for PeerDAS data columns. This enhancement allows nodes to efficiently exchange only the missing parts of data columns, reducing network overhead and accelerating the assembly of complete messages. The core logic resides in new classes that manage partial message state and integrate seamlessly with a modified Gossipsub implementation. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the partial messages extension for PeerDAS, which is a significant feature for efficient data column propagation. The changes are well-structured, with the core logic encapsulated in the new partialColumns.ts file and integrated into the existing network stack. The new functionality is also accompanied by a comprehensive set of unit tests.
My review focuses on improving code maintainability by reducing duplication in some of the new utility functions. Overall, this is a solid contribution.
| private countParts(metadata: Uint8Array): number { | ||
| let count = 0; | ||
| for (const byte of metadata) { | ||
| let b = byte; | ||
| while (b !== 0) { | ||
| count += b & 1; | ||
| b >>= 1; | ||
| } | ||
| } | ||
| return count; | ||
| } |
There was a problem hiding this comment.
This function has the same implementation as the exported function countColumnsInMetadata. To avoid code duplication, consider extracting this logic into a single, shared utility function that both can use.
Additionally, the current bit counting implementation can be optimized. Instead of iterating through each bit, you can use a technique that iterates only for each set bit, which is more efficient for sparse bitmaps.
Here's a suggested implementation using Brian Kernighan's algorithm:
function countSetBits(metadata: Uint8Array): number {
let count = 0;
for (const byte of metadata) {
let b = byte;
while (b > 0) {
// This clears the least significant set bit
b &= (b - 1);
count++;
}
}
return count;
}| export function isCompleteMetadata(metadata: Uint8Array): boolean { | ||
| const fullBytes = Math.floor(NUMBER_OF_COLUMNS / 8); | ||
| const remainingBits = NUMBER_OF_COLUMNS % 8; | ||
|
|
||
| for (let i = 0; i < fullBytes; i++) { | ||
| if (metadata[i] !== 0xff) { | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| if (remainingBits > 0) { | ||
| const expectedMask = (1 << remainingBits) - 1; | ||
| if ((metadata[fullBytes] & expectedMask) !== expectedMask) { | ||
| return false; | ||
| } | ||
| } | ||
|
|
||
| return true; | ||
| } |
There was a problem hiding this comment.
This function is identical to the private method hasAllParts in the PartialDataColumn class. To improve maintainability and reduce code duplication, this logic should be consolidated into a single utility function that both can use.
For example, you can keep this exported isCompleteMetadata function and have PartialDataColumn.hasAllParts call it.
| "@chainsafe/discv5": "^11.0.4", | ||
| "@chainsafe/enr": "^5.0.1", | ||
| "@chainsafe/libp2p-gossipsub": "^14.1.2", | ||
| "@chainsafe/libp2p-gossipsub": "file:../../../js-libp2p-gossipsub", |
raulk
force-pushed
the
feat/partial-messages-extension
branch
from
ff13ccf to
c6e771e
Compare
Integrate js-libp2p-gossipsub partial messages extension to support efficient propagation of data columns in PeerDAS. This enables nodes to exchange parts metadata and assemble complete messages from multiple partial transmissions. Key changes: - Add PartialDataColumn implementing PartialMessage interface - Add PartialColumnBroadcaster implementing PartialMessageExtension - Wire up partial message extension in Eth2Gossipsub via NetworkCore - Add bitmap encoding/decoding utilities for parts metadata - Add validation functions for partial message metadata - Include unit tests for all utility functions The extension uses a bitmap representation where each bit corresponds to a data column index (0-127 for NUMBER_OF_COLUMNS=128).
Add comprwqehensive partial messages support for efficient data column propagation. This enables nodes to exchange only the cells they're missing rather than full columns, reducing bandwidth by ~500KiB/slot. Key components: - ColumnAvailabilityStore: Track available columns per block with LRU - ReconstructionStateManager: Handle incoming partials and peer metadata - PartialColumnBroadcaster: Rebroadcast new columns to mesh peers - PartialColumnNetwork: Network layer integration with req/resp fallback - Validation: KZG proof verification for incoming partial columns Features: - Aggregated HAVE set tracking (all columns, not single) - Peer column metadata tracking for smart fetching - Automatic rebroadcast when new columns received - Req/resp fallback for missing custody columns - CLI flag: --network.partialMessages - Prometheus metrics for observability - 79 unit tests for core components
raulk
force-pushed
the
feat/partial-messages-extension
branch
from
c6e771e to
e9e7fd8
Compare
1 participant
Summary
New files
Core implementation
columnAvailability.ts- Interface for tracking which columns are available per blockcolumnAvailabilityStore.ts- In-memory LRU implementation (64 blocks, ~2 epochs TTL)reconstructionState.ts- Tracks peer column metadata and incoming partial messagespartialColumnNetwork.ts- Network layer integration with req/resp fallbackpartialColumns.ts-AggregatedPartialDataColumnandPartialColumnBroadcasterclassesValidation
chain/validation/partialDataColumn.ts- KZG proof validation for incoming partial columnsTests
columnAvailabilityStore.test.ts- 32 unit tests for column trackingreconstructionState.test.ts- 47 unit tests for peer metadata and reconstructionpartialColumns.test.ts- Unit tests for bitmap utilitiesModified files
gossipsub.ts- AddedenablePartialMessagesandpartialMessageExtensionoptionsnetworkCore.ts- Wire upPartialColumnBroadcasterin network initializationnetwork/options.ts- AddedenablePartialMessagesconfig optioncli/options/beaconNodeOptions/network.ts- Added--network.partialMessagesCLI flagmetrics/lodestar.ts- Added 5 Prometheus metrics for observabilityImplementation details
Column availability tracking
Aggregated HAVE set
partialMessageBytes()sends only columns the peer is missingReconstruction state
Rebroadcast logic
Metrics
lodestar_partial_columns_received_total- Received partials (new/duplicate/invalid)lodestar_partial_columns_rebroadcast_total- Rebroadcasts sentlodestar_partial_columns_fetched_total- Columns fetched via req/resplodestar_partial_groups_tracked- Block roots being trackedlodestar_partial_column_availability_ratio- Column availability histogramTest plan
yarn check-types)