Merged
Conversation
Introduce a dedicated WITNESS thread and supporting pipeline to offload block-witness generation and peer witness handling. Changes include: - New ThreadTypes.WITNESS and MessageType entries for witness-related messages. - New WitnessThread and WitnessThreadManager to receive forwarded block events and peer witness data, buffer early peer messages, and coordinate witness processing. - WitnessSerializer utilities to reconstruct Long instances lost by structured-clone when crossing worker boundaries (reconstructBlockWitness / reconstructSyncResponse). - BlockWitnessManager: added selfWitness queue, concurrent self-proof generation (draining logic, MAX_CONCURRENT_SELF_PROOFS), and queueSelfWitness API to decouple/async self-witness creation. - P2PManager: methods to queue self proofs, update consensus height, broadcast/request witnesses, and forward incoming peer witness messages/responses to the WITNESS thread. - PoC & PoCThread: forward BLOCK_PROCESSED events to the WITNESS thread, add APIs to request/broadcast witnesses, and reconstruct witnesses on receipt from worker threads. - Threading/Services updates: wire up the WITNESS thread in Core, PoCThreadManager, BitcoinRPCThreadManager, ServicesConfigurations and WorkerConfigurations so the thread is created and linked. - Tests: add unit tests for WitnessSerializer and WitnessThread behavior. - Minor config change: use btc.sample.conf as configPath in Config.ts. Rationale: separate heavy/IO-bound witness proof generation into its own worker to avoid blocking P2P/indexer threads, handle structured-clone degradation of Longs across worker boundaries, and allow controlled concurrency for self-generated proof work.
BlobMaster41
added
bug
Comment out creation of the ThreadTypes.WITNESS thread to prevent it from starting during initialization (likely temporary/diagnostic). Update config path to load config/btc.conf instead of the sample config/btc.sample.conf. Also apply a minor import formatting cleanup in Core.ts.
BlobMaster41
force-pushed
the
fix/thread-witness-validation
branch
from
530f72e to
3cf2f75
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a dedicated WITNESS worker thread and message pipeline to move block-witness proof generation and peer witness handling off the P2P/Indexer paths, while adding utilities to reconstruct Long values degraded by structured-clone across worker boundaries.
Changes:
- Add
ThreadTypes.WITNESSplus witness-specificMessageTypeentries and wire the new thread intoCore+ thread managers/configuration. - Implement
WitnessThread/WitnessThreadManagerandWitnessSerializerto buffer early peer messages and reconstructLongfields after cross-thread messaging. - Update witness/proof flow in
BlockWitnessManager,P2PManager,PoC, andPoCThread, and add unit tests for serializer/thread behavior.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/witness/setup.ts | Shared witness test setup (Promise.safeAll polyfill). |
| tests/witness/WitnessThread.test.ts | Unit/integration-style tests for WitnessThread message flow and forwarding behavior. |
| tests/witness/WitnessSerializer.test.ts | Tests for reconstructing Long values after structured clone. |
| src/src/threading/thread/enums/ThreadTypes.ts | Adds new WITNESS thread type. |
| src/src/threading/enum/MessageType.ts | Adds witness-specific inter-thread message types. |
| src/src/services/ServicesConfigurations.ts | Registers WITNESS thread targets and worker resource limits. |
| src/src/poc/witness/WitnessThreadManager.ts | New thread manager for WITNESS worker(s) and link setup. |
| src/src/poc/witness/WitnessThread.ts | New witness worker handling block events, buffering, and peer witness processing. |
| src/src/poc/witness/WitnessSerializer.ts | Utilities to reconstruct Long instances for witness packets crossing worker boundaries. |
| src/src/poc/peer/OPNetPeer.ts | Adjusts witness-response callback typing/behavior for the updated witness flow. |
| src/src/poc/networking/server/ServerPeerNetworking.ts | Adjusts sync-response handling to match updated witness-response callback behavior. |
| src/src/poc/networking/p2p/BlockWitnessManager.ts | Adds self-witness queue + controlled concurrency for self proof generation. |
| src/src/poc/networking/P2PManager.ts | Adds APIs for queuing/self proofs, forwarding peer witness messages to WITNESS thread, and witness broadcast/request helpers. |
| src/src/poc/PoCThreadManager.ts | Links P2P thread with the new WITNESS thread. |
| src/src/poc/PoCThread.ts | Handles witness thread requests (broadcast/request peers) and reconstructs witnesses before use. |
| src/src/poc/PoC.ts | Forwards BLOCK_PROCESSED to WITNESS thread and keeps P2P-side consensus height updated. |
| src/src/config/Config.ts | Changes default config path. |
| src/src/blockchain-indexer/rpc/BitcoinRPCThreadManager.ts | Adds link creation with WITNESS thread. |
| src/src/Core.ts | Ensures WITNESS thread is created when PoC is enabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Swap the initialization order of the WITNESS and P2P threads in Core.ts when POC is enabled so the WITNESS thread is created before the P2P thread. This ensures startup order/dependency requirements are respected during initialization.
Introduce a distinct WITNESS_HEIGHT_UPDATE message and refactor witness handling so block height is broadcast to all witness instances while proof generation is round‑robined to a single instance. PoC.onBlockProcessed is serialized via a promise lock, broadcasts WITNESS_HEIGHT_UPDATE to all witnesses, awaits it, then sends WITNESS_BLOCK_PROCESSED to one witness. WitnessThread now handles WITNESS_HEIGHT_UPDATE to set currentBlock and flush buffered peer messages; WITNESS_BLOCK_PROCESSED no longer sets height or receives an onHeightSet callback. Added handling for WITNESS messages in BitcoinRPCThread and a small comment in WitnessThreadManager. Updated tests to reflect the new message flow and behavior.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Replace busy-wait polling in BlockWitnessManager with a slotWaiters queue to await concurrency slots and wake a waiter when a slot frees, reducing CPU usage. Ensure Long values are parsed as unsigned by passing the unsigned flag to Long.fromString/fromNumber in WitnessSerializer. Add error handling for the async onBlockWitnessResponse call in WitnessThread to log rejections instead of leaving unhandled promise rejections. Update tests to await async onBlockProcessed behavior and add tests covering serialized height broadcasts and burst handling to ensure ordering and no dropped blocks.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+156
to
+159
| /** | ||
| * Replay peer witness messages that were buffered before the first | ||
| * WITNESS_BLOCK_PROCESSED set the current block height. | ||
| */ |
There was a problem hiding this comment.
The comment above flushPendingPeerMessages() refers to WITNESS_BLOCK_PROCESSED setting the current block height, but the code path that sets the height (and triggers the flush) is WITNESS_HEIGHT_UPDATE. Updating this comment will keep the buffering/flush behavior understandable and consistent with the implementation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Introduce a dedicated WITNESS thread and supporting pipeline to offload block-witness generation and peer witness handling. Changes include:
Rationale: separate heavy/IO-bound witness proof generation into its own worker to avoid blocking P2P/indexer threads, handle structured-clone degradation of Longs across worker boundaries, and allow controlled concurrency for self-generated proof work.
Type of Change
Checklist
Build & Tests
npm installcompletes without errorsnpm run buildcompletes without errorsnpm testpasses all testsCode Quality
Documentation
Security
OP_NET Node Specific
Testing
Consensus Impact
Related Issues
By submitting this PR, I confirm that my contribution is made under the terms of the project's license.