diff --git a/Agents.md b/Agents.md index 7693af10a..9a97141cb 100644 --- a/Agents.md +++ b/Agents.md @@ -32,4 +32,5 @@ pnpm run release:rc These scripts forward to aegir with the pnpm publish command, so no additional flags are required. - +General Instructions +- For any code exploration spawn subagents for the task \ No newline at end of file diff --git a/docs/architecture-viz-phase2.html b/docs/architecture-viz-phase2.html new file mode 100644 index 000000000..f93fe1c36 --- /dev/null +++ b/docs/architecture-viz-phase2.html @@ -0,0 +1,3304 @@ + + + + + +Peerbit Architecture Visualization (Phase 2) + + + + + + +
+ Peerbit Architecture (Phase 2) + +
+
+ + Transport + libp2p +
+
+ + PubSub + Peerbit +
+
+ + Blocks + Peerbit +
+
+ + Ranges + Peerbit +
+
+ + Entries + Peerbit +
+
+ +
+ + + + +
+ + +
+ + +
+
+ Event Log + +
+
+
+ + + +
+ +
+ + +
Click a node to connect to — press Esc to cancel
+ + +
+
+ + +
+ + +
+ + +
+
+ + + + + +
+ +
+ Events: 0 +
+
+
+ +
+
Ready — right-click the canvas to add nodes or use the button above
+
+ + +
+
+ +

Peerbit Architecture Visualization (Phase 2)

+

Interactive visualization of Peerbit's P2P database architecture showing data flow across all layers.

+

Mouse Controls

+ +

Keyboard Shortcuts

+ +

Architecture Layers

+ +

Peerbit Data Flow

+

When peers connect: Transport → PubSub/Blocks negotiate in parallel → (both ready) Exchange heads.

+

When appending: entries are sent directly to the leader replicators whose ranges cover the entry coordinate.

+

SharedLog entries are assigned coordinates in a circular 0..1 space. Each peer's replication range determines which entries it stores.

+
+
+ + +
+
+

Import State

+

Upload a JSON file or paste exported JSON below.

+ + + +
+ + +
+
+
+ + + + diff --git a/docs/architecture-viz.html b/docs/architecture-viz.html new file mode 100644 index 000000000..e360fe563 --- /dev/null +++ b/docs/architecture-viz.html @@ -0,0 +1,3029 @@ + + + + + +Peerbit Architecture Visualization + + + + + + +
+ Peerbit Architecture + +
+
+ + Transport +
+
+ + PubSub +
+
+ + Blocks +
+
+ + Ranges +
+
+ + Entries +
+
+ +
+ + + + +
+ + +
+ + +
+
+ Event Log + +
+
+
+ + + +
+ +
+ + +
Click a node to connect to — press Esc to cancel
+ + +
+
+ + +
+ + +
+
+ + + + + +
+ +
+ Events: 0 +
+
+
+ +
+
Ready — right-click the canvas to add nodes or use the button above
+
+ + +
+
+ +

Peerbit Architecture Visualization

+

Interactive visualization of Peerbit's P2P database architecture showing data flow across all layers.

+

Mouse Controls

+ +

Keyboard Shortcuts

+ +

Architecture Layers

+ +

Peerbit Data Flow

+

When peers connect: Transport → PubSub ready → Blocks ready → Exchange heads → Replicate entries.

+

SharedLog entries are assigned coordinates in a circular 0..1 space. Each peer's replication range determines which entries it stores.

+
+
+ + +
+
+

Import State

+

Upload a JSON file or paste exported JSON below.

+ + + +
+ + +
+
+
+ + + + diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md new file mode 100644 index 000000000..c4455f973 --- /dev/null +++ b/docs/implementation-plan.md @@ -0,0 +1,189 @@ +# Architecture Visualization — Implementation Plan (Phase 2) + +Findings from comparing `docs/architecture-viz.html` against the real Peerbit codebase. +Goal: bring the viz closer to the actual system behavior. + +--- + +## Accuracy Audit + +### What the viz gets right +- Layered architecture: transport, pubsub, blocks +- Programs with shared logs subscribing to topics for peer discovery +- Replication ranges in circular coordinate space with wraparound +- Bidirectional information exchange when peers connect +- Documents as put/delete operations on top of SharedLog +- DAG-linked entries with heads tracking + +### Critical inaccuracies to fix + +| # | Issue | Viz behavior | Real behavior | Priority | +|---|-------|-------------|---------------|----------| +| 1 | Replication targeting | Floods entries to all connected peers with same program | Leader-based targeted delivery — entries sent only to replicators whose ranges cover the entry's coordinates | **High** | +| 2 | Cascading replication | Peer relays to its other peers (gossip) | Originator sends directly to all relevant leaders; no relay chain | **High** | +| 3 | Connection cascade | Sequential: transport → pubsub → blocks with fixed offsets | Pubsub and blocks are independent protocols negotiating in parallel | **Medium** | +| 4 | Program addressing | Random string (`program-`) | Content-addressed CID from serialized program | **Low** | +| 5 | Range space | Float `[0, 1]` | Integer u32 `[0, 4294967295]` or u64 | **Low** (conceptually equivalent) | + +--- + +## Implementation Tasks + +### Phase 2a — Fix replication model (High priority) + +#### 2a.1 Leader-based entry delivery +Replace the current `replicateEntry()` flood with leader computation: +- When appending an entry, compute its coordinate (already exists) +- Find which replicators' ranges cover that coordinate (`rangeContains()` already exists) +- Send the entry only to those leaders, not to all connected peers +- Update `replicateEntry(sourceNodeId, programAddress, entry)` to: + 1. Collect all replication ranges across all nodes for the program + 2. Filter to ranges that contain `entry.coordinate` + 3. Send only to nodes owning those ranges (that are reachable via connections) +- Add event type `replication:leader-send` to distinguish from generic replication + +#### 2a.2 Remove cascading replication +- Delete the cascading relay logic in `replicateEntry()` (the inner loop at lines 951-964) +- Delete `replicateEntryDirect()` (only used for cascading) +- Instead, have the originator iterate all known leaders and send directly +- If a leader is not directly connected, show a "no route" indicator or skip (defer multi-hop to Phase 2c) + +#### 2a.3 Visualize leader selection +- When an entry is appended, briefly highlight the leaders on the canvas (flash their replication arcs) +- Show particle animations only to the targeted leaders, not all peers +- Add a tooltip or event description showing why each leader was chosen ("range [0.2..0.5] covers coordinate 0.35") + +### Phase 2b — Fix connection model (Medium priority) + +#### 2b.1 Parallel protocol negotiation +- Change `addConnection()` so pubsub and blocks negotiate independently (not sequentially) +- Both start after transport is established, with independent random latencies +- Replace the fixed `+30ms` / `+60ms` offsets with two independent timers starting from transport-ready +- Head exchange triggers when **both** pubsub and blocks are ready (not just blocks) + +#### 2b.2 Connection state display +- Show protocol negotiation status on connection hover: "transport: ready, pubsub: negotiating, blocks: ready" +- Dashed line segments during negotiation, solid when fully connected + +### Phase 2c — Add missing systems (Lower priority) + +#### 2c.1 RPC layer abstraction +- Add visual indication that messages go through an RPC layer +- Show message types in particles: `ExchangeHeadsMessage`, `AllReplicatingSegmentsMessage`, `RequestReplicationInfoMessage` +- Color-code particles by message type + +#### 2c.2 Synchronizer (simplified) +- After initial head exchange, run a periodic "sync check" between connected peers +- Compare entry sets and request missing entries +- Show sync status in inspector: "synced with: [Alice, Bob]", "pending: [Carol]" +- Event types: `sync:request`, `sync:complete` + +#### 2c.3 Pruning protocol +- If a node holds an entry outside its replication range, schedule a prune check +- Send prune request to peers: "Can I drop entry X? Do you have it?" +- If confirmed, remove entry from local store +- Event types: `prune:request`, `prune:confirm`, `prune:drop` +- Show pruned entries fading out in the inspector + +#### 2c.4 Adaptive replication +- Add a "load" simulation to nodes (memory/CPU proxy) +- Replication range width adjusts based on load: high load → narrower range, low load → wider range +- Animate range arc width changes +- Show the PID controller concept in inspector: "target width: 0.33, current: 0.28, adjusting..." + +#### 2c.5 Multi-hop routing +- If a leader is not directly connected but reachable via intermediate nodes, route through them +- Show multi-hop particle paths traversing intermediate connections +- Event type: `route:relay` + +### Phase 2d — Polish (Low priority) + +#### 2d.1 Content-addressed program IDs +- Generate program addresses as CID-like hashes instead of random strings +- Show in inspector: "address: bafy..." + +#### 2d.2 Entry types +- Support `APPEND` and `CUT` entry types +- `CUT` entries render with strikethrough in inspector +- Document delete creates a `CUT` entry instead of a regular append + +#### 2d.3 Delivery modes +- Add delivery mode selection in context menu: Acknowledge / Silent / Seek +- Acknowledge mode: show return particle (ACK) from leader back to sender +- Silent mode: no ACK particle + +#### 2d.4 GID grouping +- Group entries by GID in inspector view +- Show GID-based leader computation: "GID g-abc123 → leaders: [Alice, Bob]" + +--- + +## Files to modify +- `docs/architecture-viz.html` — baseline (Phase 1), keep unchanged +- `docs/architecture-viz-phase2.html` — Phase 2 parallel implementation (all Phase 2 changes go here) + +## Key functions to change +- `engine.replicateEntry()` — replace flood with leader targeting (2a.1, 2a.2) +- `engine.replicateEntryDirect()` — remove (2a.2) +- `engine.addConnection()` — parallel protocol negotiation (2b.1) +- `engine.exchangeHeadsIfNeeded()` — add replication segment exchange (2c.1) +- `engine.appendEntry()` — add leader computation before replication (2a.1) +- `renderer.drawParticle()` — message type labels (2c.1) +- `renderer.drawNode()` — leader highlight flash (2a.3) +- New: `engine.findLeaders(coordinate, programAddress)` — compute leaders from ranges (2a.1) +- New: `engine.syncCheck(nodeIdA, nodeIdB)` — simplified synchronizer (2c.2) +- New: `engine.pruneEntry(nodeId, entryHash)` — pruning protocol (2c.3) + +--- + +## Reference: Real implementation files +- `packages/clients/peerbit/src/peer.ts` — `Peerbit.dial()`, `open()` +- `packages/transport/stream/src/index.ts` — `DirectStream`, connection lifecycle +- `packages/transport/pubsub/src/index.ts` — `DirectSub`, subscriptions +- `packages/transport/blocks/src/libp2p.ts` — `DirectBlock` +- `packages/programs/program/program/src/program.ts` — `Program` base class lifecycle +- `packages/programs/data/shared-log/src/index.ts` — `SharedLog.append()`, `handleSubscriptionChange()`, `onMessage()` +- `packages/programs/data/shared-log/src/exchange-heads.ts` — `ExchangeHeadsMessage` +- `packages/programs/data/shared-log/src/ranges.ts` — `ReplicationRangeIndexable`, segment math +- `packages/programs/data/shared-log/src/replication.ts` — `MinReplicas`, replication messages +- `packages/programs/data/document/document/src/program.ts` — `Documents.put()`, `del()` + +--- + +## Parallel Implementation Artifact + +Baseline (Phase 1): `docs/architecture-viz.html` (do not edit) + +Parallel implementation (Phase 2): `docs/architecture-viz-phase2.html` + +--- + +## Running Log + +### Learnings +- 2026-02-08: In `docs/architecture-viz.html`, `exchangeHeadsIfNeeded()` currently syncs missing entries by directly calling `replicateEntryDirect()` (so `replicateEntryDirect()` is not only used for cascading replication today). +- 2026-02-08: `openProgram()` triggers head exchange against any connection with `conn.state === 'connected'` (set at transport-ready), even when `pubsub`/`blocks` negotiation is not yet complete. +- 2026-02-08: Connection visualization currently only differentiates `conn.state` (`connecting` vs `connected`) and partially uses `conn.layers.blocks` for alpha; there is no existing connection hover/tooltip system. +- 2026-02-08: Timeline scrubbing (`timeline.seekTo`) restores snapshots and only re-spawns animations for a small subset of event types; new “leader send” visuals need explicit handling so they show up while stepping/scrubbing. +- 2026-02-08: In the real code, “pubsub” (`DirectSub`) and “blocks” (`DirectBlock`) are Peerbit implementations registered as libp2p services (they run *over* libp2p connections/streams, but are not libp2p’s gossipsub/bitswap). +- 2026-02-08: The layer toggle UI benefits from explicitly labeling “libp2p” vs “Peerbit” ownership to avoid confusing Peerbit’s DirectSub/DirectBlock with libp2p gossipsub/bitswap. + +### Ahas +- 2026-02-08: The cleanest “parallel implementation” is a new HTML file that starts as a copy of `docs/architecture-viz.html`, so Phase 2 changes can iterate without breaking the baseline demo. +- 2026-02-08: To make leader-selection visuals actually visible during timeline stepping, hook the effects to `replication:leader-send` in `timeline.seekTo` (similar to how `replication:entry-replicate` spawns particles today). + +### Answers To Questions +- 2026-02-08: “Do not edit the existing one” interpreted as: keep `docs/architecture-viz.html` unchanged; implement Phase 2 behaviors in a new `docs/architecture-viz-phase2.html` file. +- 2026-02-08: Layer ownership (what is Peerbit vs libp2p?): + - Transport: libp2p connection + stream mux/encryption (Peerbit uses libp2p underneath) + - PubSub: Peerbit `DirectSub` (`packages/transport/pubsub/...`) running over libp2p streams + - Blocks: Peerbit `DirectBlock` (`packages/transport/blocks/...`) running over libp2p streams + - Entries: Peerbit log entries (`packages/log/...`) stored as content-addressed blocks + - Ranges: Peerbit SharedLog replication ranges (`packages/programs/data/shared-log/src/ranges.ts`) + +### Next Steps +- Done: Create `docs/architecture-viz-phase2.html` by copying the baseline file. +- Done: Phase 2a leader-based replication targeting (and remove cascading replication) implemented in `docs/architecture-viz-phase2.html`. +- Done: Phase 2b connection protocol negotiation in parallel (and gate head exchange on both pubsub+blocks ready) implemented in `docs/architecture-viz-phase2.html`. +- Done: Leader-selection visualization (range arc flash), `replication:leader-send` event, and connection hover status tooltip implemented in `docs/architecture-viz-phase2.html`. +- Next: Implement Phase 2c.1 (RPC message types) by tagging particles/events with message-type labels and updating the particle renderer accordingly. diff --git a/docs/test-timeline-playwright.mjs b/docs/test-timeline-playwright.mjs new file mode 100644 index 000000000..bab0d99f4 --- /dev/null +++ b/docs/test-timeline-playwright.mjs @@ -0,0 +1,178 @@ +/** + * Playwright test for timeline slider state replay. + * Run: node docs/test-timeline-playwright.mjs + */ +import { chromium } from '@playwright/test'; +import { fileURLToPath } from 'url'; +import { dirname, resolve } from 'path'; + +const __filename = fileURLToPath(import.meta.url); +const __dirname = dirname(__filename); + +let failures = 0; +function assert(cond, msg) { + if (!cond) { console.error('FAIL:', msg); failures++; } + else { console.log(' OK:', msg); } +} + +async function run() { + const browser = await chromium.launch({ headless: true }); + const page = await browser.newPage(); + + // Collect console errors + const consoleErrors = []; + page.on('console', msg => { + if (msg.type() === 'error') consoleErrors.push(msg.text()); + }); + page.on('pageerror', err => { + consoleErrors.push(err.message); + }); + + const filePath = resolve(__dirname, 'architecture-viz.html'); + await page.goto('file://' + filePath); + await page.waitForTimeout(500); + + // --- Test: Page loads without JS errors --- + console.log('\n--- Test: Page loads without errors ---'); + assert(consoleErrors.length === 0, 'No console errors on load: ' + (consoleErrors.length > 0 ? consoleErrors.join('; ') : 'clean')); + + // --- Load scenario --- + console.log('\n--- Test: Load scenario ---'); + const options = await page.evaluate(() => { + const sel = document.getElementById('scenario-select'); + return [...sel.options].map((o, i) => ({ index: i, value: o.value, text: o.text })); + }); + console.log(' Scenarios:', options.map(o => o.text).join(', ')); + + if (options.length > 1) { + await page.selectOption('#scenario-select', options[1].value); + } + await page.waitForTimeout(3000); // Wait for scenario events + + const eventCount = await page.evaluate(() => engine.eventLog.length); + console.log(' Events after scenario:', eventCount); + assert(eventCount > 3, 'Scenario generated events (got ' + eventCount + ')'); + + const stateHistoryLen = await page.evaluate(() => engine.stateHistory.length); + console.log(' stateHistory.length:', stateHistoryLen); + assert(stateHistoryLen === eventCount, 'stateHistory.length (' + stateHistoryLen + ') matches eventLog.length (' + eventCount + ')'); + + // --- Test: Slider max --- + console.log('\n--- Test: Slider max matches events ---'); + const sliderMax = await page.evaluate(() => +document.getElementById('timeline-slider').max); + assert(sliderMax === eventCount - 1, 'slider.max (' + sliderMax + ') === eventCount-1 (' + (eventCount - 1) + ')'); + + // --- Record full state for comparison --- + const fullNodeCount = await page.evaluate(() => engine.nodes.size); + const fullConnCount = await page.evaluate(() => engine.connections.size); + console.log(' Full state: ' + fullNodeCount + ' nodes, ' + fullConnCount + ' connections'); + + // --- Test: seekTo(0) --- + console.log('\n--- Test: seekTo(0) restores first event state ---'); + consoleErrors.length = 0; + + await page.evaluate(() => timeline.seekTo(0)); + + const seek0Errors = [...consoleErrors]; + const seek0Nodes = await page.evaluate(() => engine.nodes.size); + const seek0Conns = await page.evaluate(() => engine.connections.size); + const seek0Counter = await page.evaluate(() => document.getElementById('event-counter').textContent); + const seek0Desc = await page.evaluate(() => document.getElementById('event-description').textContent); + const seek0Index = await page.evaluate(() => timeline.currentIndex); + + console.log(' Errors:', seek0Errors.length > 0 ? seek0Errors.join('; ') : 'none'); + console.log(' currentIndex:', seek0Index); + console.log(' Counter text:', seek0Counter); + console.log(' Description:', seek0Desc); + console.log(' Nodes:', seek0Nodes, ' Connections:', seek0Conns); + + assert(seek0Errors.length === 0, 'No errors during seekTo(0)'); + assert(seek0Index === 0, 'currentIndex is 0'); + assert(seek0Counter.includes('1 /'), 'Counter shows event 1 (got: ' + seek0Counter + ')'); + assert(seek0Desc.length > 5, 'Description is not empty'); + assert(seek0Nodes <= 1, 'At event 0, at most 1 node (got ' + seek0Nodes + ')'); + assert(seek0Conns === 0, 'At event 0, 0 connections (got ' + seek0Conns + ')'); + + // --- Test: seekTo(end) restores full state --- + console.log('\n--- Test: seekTo(end) restores full state ---'); + consoleErrors.length = 0; + + await page.evaluate(() => timeline.seekTo(engine.eventLog.length - 1)); + + const endNodes = await page.evaluate(() => engine.nodes.size); + const endConns = await page.evaluate(() => engine.connections.size); + + assert(endNodes === fullNodeCount, 'End nodes (' + endNodes + ') === full (' + fullNodeCount + ')'); + assert(endConns === fullConnCount, 'End conns (' + endConns + ') === full (' + fullConnCount + ')'); + assert(consoleErrors.length === 0, 'No errors during seekTo(end)'); + + // --- Test: Slider input event --- + console.log('\n--- Test: Slider oninput triggers seekTo ---'); + consoleErrors.length = 0; + + await page.evaluate(() => { + const slider = document.getElementById('timeline-slider'); + slider.value = '0'; + slider.dispatchEvent(new Event('input')); + }); + + const inputIndex = await page.evaluate(() => timeline.currentIndex); + const inputNodes = await page.evaluate(() => engine.nodes.size); + const inputCounter = await page.evaluate(() => document.getElementById('event-counter').textContent); + + assert(inputIndex === 0, 'Slider input set currentIndex to 0 (got ' + inputIndex + ')'); + assert(inputNodes <= 1, 'Slider input restored state (nodes=' + inputNodes + ')'); + assert(inputCounter.includes('1 /'), 'Counter updated from slider (got: ' + inputCounter + ')'); + assert(consoleErrors.length === 0, 'No errors from slider input: ' + consoleErrors.join('; ')); + + // --- Test: stepForward / stepBack --- + console.log('\n--- Test: Step forward/back ---'); + await page.evaluate(() => timeline.seekTo(0)); + await page.evaluate(() => timeline.stepForward()); + const fwdIndex = await page.evaluate(() => timeline.currentIndex); + assert(fwdIndex === 1, 'stepForward → index 1 (got ' + fwdIndex + ')'); + + await page.evaluate(() => timeline.stepBack()); + const backIndex = await page.evaluate(() => timeline.currentIndex); + assert(backIndex === 0, 'stepBack → index 0 (got ' + backIndex + ')'); + + // --- Test: Sidebar highlighting --- + console.log('\n--- Test: Sidebar event highlighting ---'); + await page.evaluate(() => timeline.seekTo(3)); + const activeCount = await page.evaluate(() => document.querySelectorAll('.event-row.active').length); + const activeIdx = await page.evaluate(() => { + const rows = document.getElementById('event-list').children; + for (let i = 0; i < rows.length; i++) { + if (rows[i].classList.contains('active')) return i; + } + return -1; + }); + assert(activeCount === 1, 'Exactly 1 active event row (got ' + activeCount + ')'); + assert(activeIdx === 3, 'Active row is index 3 (got ' + activeIdx + ')'); + + // --- Test: State increases monotonically through events --- + console.log('\n--- Test: Node count increases through early events ---'); + let prevNodes = 0; + let monotonicOk = true; + for (let i = 0; i < Math.min(eventCount, 5); i++) { + await page.evaluate((idx) => timeline.seekTo(idx), i); + const n = await page.evaluate(() => engine.nodes.size); + if (n < prevNodes) { monotonicOk = false; console.log(' Event ' + i + ': nodes=' + n + ' < prev=' + prevNodes); } + prevNodes = n; + } + assert(monotonicOk, 'Node count is non-decreasing in first 5 events'); + + // --- Final error check --- + console.log('\n--- Final error check ---'); + assert(consoleErrors.length === 0, 'No accumulated JS errors'); + + await browser.close(); + + console.log('\n' + (failures === 0 ? 'ALL TESTS PASSED' : `${failures} FAILURE(S)`)); + process.exit(failures === 0 ? 0 : 1); +} + +run().catch(e => { + console.error('Test runner error:', e); + process.exit(1); +}); diff --git a/implementations/001-architecture-diagram-implementation-plan.md b/implementations/001-architecture-diagram-implementation-plan.md new file mode 100644 index 000000000..5743b521c --- /dev/null +++ b/implementations/001-architecture-diagram-implementation-plan.md @@ -0,0 +1,326 @@ +Peerbit Architecture Visualization Tool - Implementation Plan + + Context + + Peerbit is a complex P2P database framework with many layers (libp2p transport, pubsub, blocks, programs, SharedLog, document store, + replication ranges). Newcomers struggle to understand the data flow and architecture. This tool will be an interactive, animated + visualization that lets users create peer nodes, connect them, open programs, append entries, and watch replication happen in real time — + with a step-through timeline and React-DevTools-like state inspectors. + + Decisions Made + Decision: Simulation model + Choice: Hybrid — pure simulation engine now, designed so data model can later be backed by real Peerbit nodes + ──────────────────────────────────────── + Decision: Scope + Choice: Full stack — all layers (transport, pubsub, blocks, programs, SharedLog, documents, replication) with toggleable visibility + ──────────────────────────────────────── + Decision: Deployment + Choice: Standalone HTML file — self-contained, CDN dependencies only + ──────────────────────────────────────── + Decision: Renderer + Choice: D3 + Canvas hybrid — Canvas for network graph, HTML/CSS for inspector panels and UI + ──────────────────────────────────────── + Decision: Inspector + Choice: Floating panels per node — multiple open simultaneously + ──────────────────────────────────────── + Decision: Actions + Choice: Context menu + action bar — right-click nodes/connections, global buttons + ──────────────────────────────────────── + Decision: Timeline + Choice: Event-based steps — discrete events, play/pause/step, speed control + File Location + + /Users/aristotle/Documents/Projects/peerbit/docs/architecture-viz.html + + Single HTML file (~3,600 lines). Only external dependency: D3.js v7 from CDN. + + Architecture Overview + + ┌──────────────────────────────────────────────────────┐ + │ HTML/CSS Shell │ + │ ┌─────────────┐ ┌────────────────────────────────┐ │ + │ │ Action Bar │ │ Canvas Container │ │ + │ │ (top) │ │ │ │ + │ │ [+Node] │ │ Canvas: nodes, connections, │ │ + │ │ [Scenario] │ │ particles, replication arcs │ │ + │ │ [Layers] │ │ │ │ + │ └─────────────┘ │ HTML Overlay: floating │ │ + │ │ inspector panels per node │ │ + │ └────────────────────────────────┘ │ + │ ┌────────────────────────────────────────────────┐ │ + │ │ Timeline Panel (bottom) │ │ + │ │ |< < [▶] > >| [1x] [2x] ───●────────── │ │ + │ │ Event: "Node A: entry appended [hash: e005]" │ │ + │ └────────────────────────────────────────────────┘ │ + └──────────────────────────────────────────────────────┘ + │ │ + ▼ ▼ + ┌─────────────────┐ ┌─────────────────┐ + │ SimulationEngine │ │ CanvasRenderer │ + │ (state machine) │ │ (D3 + Canvas) │ + │ │ │ │ + │ - nodes │ │ - force layout │ + │ - connections │ │ - node circles │ + │ - programs │ │ - connection lines│ + │ - sharedLogs │ │ - particles │ + │ - eventLog[] │ │ - range arcs │ + │ - pendingEvents │ │ - hit testing │ + └─────────────────┘ └─────────────────┘ + + Data Model (Key Interfaces) + + Modeled directly from the Peerbit codebase: + + SimNode (mirrors Peerbit class in peer.ts) + ├── identity: { peerId, publicKeyHash, displayName, color } + ├── programs: Map + ├── connections: Set + ├── pubsub: { subscriptions, topicSubscribers } + ├── blockStore: { blocks, totalSize } + ├── position: { x, y } + └── status: 'active' | 'stopping' | 'stopped' + + SimProgram (mirrors Program in program.ts) + ├── address, type, name, closed + ├── sharedLogs: Map + ├── topics: string[] + └── children: string[] + + SimSharedLog (mirrors SharedLog in shared-log/src/index.ts) + ├── entries: Map + ├── heads: Set + ├── replicationRanges: Map + ├── replicas: { min, max } + └── syncState: { pendingSync, syncedWith } + + SimEntry (mirrors Entry in log/src/entry.ts) + ├── hash, gid, data, creatorPeerId + ├── clock: { timestamp, counter } + ├── next: string[] (DAG links) + ├── coordinate: number (0..1, position in replication space) + └── replicatedBy: Set + + SimReplicationRange (mirrors ReplicationRangeIndexableU32 in ranges.ts) + ├── offset: number (0..1) + ├── width: number (0..1) + ├── mode: 'strict' | 'non-strict' + └── matured: boolean + + SimConnection (mirrors libp2p connection + service readiness) + ├── fromPeerId, toPeerId + ├── state: 'connecting' | 'connected' | 'disconnecting' + ├── layers: { transport, pubsub, blocks } + └── latencyMs, bandwidth + + Event System + + Every state change is a discrete event on the timeline: + + Event types (grouped by layer): + - Transport: node:create, node:destroy, connection:dial, connection:established, connection:hangup + - PubSub: pubsub:subscribe, pubsub:unsubscribe, pubsub:message, connection:pubsub-ready + - Blocks: block:request, block:deliver, connection:blocks-ready + - Program: program:open, program:close, program:drop + - SharedLog: sharedlog:append, sharedlog:exchange-heads, sharedlog:sync-request, sharedlog:sync-complete + - Replication: replication:range-announce, replication:entry-replicate, replication:prune, replicator:join, replicator:leave, + replicator:mature + - Document: document:put, document:delete + - RPC: rpc:request, rpc:response + + Cascading events (modeled from actual Peerbit behavior): + User clicks "Connect A to B" + → connection:dial + → (after latency) connection:established (transport layer up) + → connection:pubsub-ready (DirectSub streams) + → connection:blocks-ready (DirectBlock streams) + → IF both nodes share a program: + → sharedlog:exchange-heads (exchange log heads) + → FOR each missing entry: + → replication:entry-replicate (sync entry) + → replicator:join (new replicator announced) + + Timeline scrubbing: Checkpoint-and-replay strategy. Full state snapshot every 50 events. Rewind replays from nearest checkpoint. + + Canvas Rendering Strategy + + - Canvas handles: node circles, connection lines, animated particles, replication range arcs, labels + - HTML overlay handles: inspector panels, context menus, tooltips, action bar, timeline + - D3 handles: force-directed layout only (positions). Drawing is native Canvas 2D API. + - Hit testing: Simple distance-to-node check (sufficient for 10-30 nodes) + - Pan/zoom: Canvas transform matrix via ctx.translate() + ctx.scale() + + Visual encoding: + - Transport connections: solid blue lines (#4a9eff) + - PubSub traffic: animated green dots (#00cc88) + - Block transfers: animated orange dots (#ff8844) + - Replication ranges: colored arcs around node circles + - Wrapped ranges (offset+width > 1.0): two arc segments per getSegmentsFromOffsetAndRange() in ranges.ts + + Implementation Phases + + Phase 1: Core Foundation (~1000 lines) + + - HTML skeleton, CSS dark theme, all container elements + - SimulationState, SimNode, SimConnection data structures + - SimulationEngine core: addNode(), removeNode(), addConnection(), removeConnection() + - CanvasRenderer: render nodes as circles, connections as lines + - D3 force layout (auto-position nodes) + - Mouse interaction: drag nodes, pan, zoom + - Action bar: "Add Node" button + - Basic context menu: right-click node → Connect to..., Destroy + - Event log array + + Phase 2: Programs & SharedLog (~800 lines) + + - SimProgram, SimSharedLog, SimEntry data structures + - Context menu: "Open Program" → SharedLog / Documents + - Program open event cascade (create log, subscribe to topic) + - "Append Entry" action with coordinate assignment + - Basic replication: connected peers with same program auto-sync entries + - exchangeHeads simulation on connection + - Replication range data structures (offset, width, wrapping) + - Canvas: replication range arcs around nodes + - Event cascading system (connect → exchange heads → replicate) + + Phase 3: Inspector & Timeline (~800 lines) + + - Floating inspector panel (draggable, multiple simultaneous) + - Recursive collapsible tree renderer for node state + - Auto-update on events with highlight flash animation + - Timeline slider (HTML range input + custom track rendering) + - Event markers on timeline (color-coded by layer) + - Play/pause/step-forward/step-backward controls + - Speed control (0.5x, 1x, 2x, 5x) + - State checkpoint system for efficient scrubbing + - Event description display + - Toast notifications for events + + Phase 4: Advanced Visualization (~600 lines) + + - Animated particles along connections (speed, direction, color by layer) + - Layer visibility toggle buttons in action bar + - Layer-based rendering: show/hide transport, pubsub, blocks, ranges, entries + - Range arc grow-in animations + - Connection styling (dashed=connecting, solid=connected) + - Node badges (program count, entry count) + - Glow effects for active nodes + - Entry replication animation (particle from source → target) + - Simplified PID controller simulation for dynamic range adjustment + + Phase 5: Documents, RPC & Scenarios (~400 lines) + + - Document store simulation (PutOperation/DeleteOperation wrapping SharedLog) + - "Put Document" / "Delete Document" context menu actions + - RPC request/response visualization + - Pre-built scenarios: + - "3-Node Replication" — 3 nodes, connected, shared program, entries replicating + - "Dynamic Sharding" — 5 nodes, ranges rebalancing as nodes join/leave + - "Entry Lifecycle" — append, replicate, prune + - "Load Scenario" dropdown in action bar + - Keyboard shortcuts (Space=play/pause, arrows=step, N=new node, ?=help) + - Help overlay with architecture guide + + Key Codebase References + Concept: Peer client API + Source File: packages/clients/peerbit/src/peer.ts + Key Lines/Details: dial() (lines 296-325), open(), hangUp() + ──────────────────────────────────────── + Concept: libp2p setup + Source File: packages/clients/peerbit/src/libp2p.ts + Key Lines/Details: DirectSub, DirectBlock, Noise, Yamux services + ──────────────────────────────────────── + Concept: Program base + Source File: packages/programs/program/program/src/program.ts + Key Lines/Details: Lifecycle: beforeOpen(), open(), afterOpen(), close() + ──────────────────────────────────────── + Concept: Program handler + Source File: packages/programs/program/program/src/handler.ts + Key Lines/Details: items Map, open/close lifecycle management + ──────────────────────────────────────── + Concept: SharedLog + Source File: packages/programs/data/shared-log/src/index.ts + Key Lines/Details: State (lines 446-586), append() (1509-1570), topic = log.idString (line 2335) + ──────────────────────────────────────── + Concept: Replication ranges + Source File: packages/programs/data/shared-log/src/ranges.ts + Key Lines/Details: ReplicationRangeIndexableU32 (lines 664-800), wrapping logic, getSegmentsFromOffsetAndRange() (lines 47-67) + ──────────────────────────────────────── + Concept: Log & Entry + Source File: packages/log/src/log.ts, packages/log/src/entry.ts + Key Lines/Details: Entry structure: hash, meta.gid, meta.clock, meta.next, payload + ──────────────────────────────────────── + Concept: Change events + Source File: packages/log/src/change.ts + Key Lines/Details: Change = { added, removed } + ──────────────────────────────────────── + Concept: SharedLog events + Source File: packages/programs/data/shared-log/test/events.spec.ts + Key Lines/Details: replicator:join, replicator:leave, replicator:mature + ──────────────────────────────────────── + Concept: PubSub + Source File: packages/transport/pubsub/src/index.ts + Key Lines/Details: topics, peerToTopic, topicsToPeers maps + ──────────────────────────────────────── + Concept: Documents + Source File: packages/programs/data/document/document/src/program.ts + Key Lines/Details: Wraps SharedLog with PutOperation/DeleteOperation + ──────────────────────────────────────── + Concept: Bootstrap + Source File: packages/clients/peerbit/src/bootstrap.ts + Key Lines/Details: resolveBootstrapAddresses() + Verification Plan + + 1. Open the file: open docs/architecture-viz.html in a modern browser (Chrome/Firefox/Safari) + 2. Add nodes: Click "+ Add Node" 3 times. Verify nodes appear and auto-arrange via D3 force + 3. Connect nodes: Right-click Node A → "Connect to..." → click Node B. Verify connection line appears with 3-phase animation (transport → + pubsub → blocks) + 4. Open program: Right-click Node A → "Open Program" → "SharedLog". Verify program appears in inspector + 5. Append entry: Right-click Node A → "Append Entry". Verify entry appears in inspector and on the timeline + 6. Replication: Open same program on Node B. Verify entries replicate (particle animation from A to B, entry appears in B's inspector) + 7. Timeline: Use slider to scrub backwards. Verify state reverts correctly. Step forward one event at a time. + 8. Layer toggles: Toggle off "transport" layer. Verify connection lines hide but nodes remain + 9. Inspector: Open inspectors for 2 nodes simultaneously. Verify both update in real-time + 10. Scenarios: Load "3-Node Replication" scenario. Verify pre-built nodes, connections, and programs set up correctly + +## Implementation Results + +**File:** `/docs/architecture-viz.html` — 2,168 lines, single self-contained HTML file with only D3.js v7 as external dependency. + +### Features Implemented + +**Core Engine:** +- `SimulationEngine` state machine with nodes, connections, programs, shared logs, entries +- Discrete event system with 20+ event types matching Peerbit's actual architecture +- Cascading events (connect → transport → pubsub → blocks → exchange heads → replicate) + +**Canvas Rendering:** +- D3 force-directed layout for auto-positioning nodes +- Canvas 2D drawing for nodes, connections, particles, replication arcs +- Pan, zoom, drag interactions +- Animated particles along connections (color-coded by layer) +- Replication range arcs with wraparound support (mirrors `getSegmentsFromOffsetAndRange()`) +- Node badges showing program/entry counts, glow effects + +**Inspector Panels:** +- Floating, draggable panels per node +- Recursive collapsible tree view showing full node state +- Real-time updates on events + +**Timeline:** +- Event-based slider with play/pause/step-forward/step-backward +- Speed control (0.5x, 1x, 2x, 5x) +- Color-coded event markers +- Event description display + +**Interactions:** +- Right-click context menus for nodes, connections, and canvas +- Actions: Connect, Open SharedLog/Documents, Append Entry, Put/Delete Document, Disconnect, Destroy +- Connection mode with visual indicator +- Layer visibility toggles (Transport, PubSub, Blocks, Ranges, Entries) + +**Scenarios:** +- **3-Node Replication** — 3 connected nodes sharing a program with replicating entries +- **Dynamic Sharding** — 5 nodes in a ring with cross-links, each contributing to sharding +- **Entry Lifecycle** — 2 nodes showing DAG-linked entries replicating + +**Keyboard Shortcuts:** `N` (add node), `Space` (play/pause), arrows (step), `?` (help), `Esc` (cancel), `+/-` (zoom) \ No newline at end of file