Scalable fanout pubsub over @peerbit/stream (tree + pull-repair + incentives)

[marcus-pousette]

Goal

Ship a pubsub fanout solution that can support very large audiences (target: 1 publisher → 1,000,000 subscribers) with bounded per-node upload, low latency, and measurable reliability — without requiring global membership knowledge.

Tracking PR: #582

Motivation

Current pubsub subscriber discovery can “explode” because the control-plane scales poorly (subscription gossip amplification). At large scale, the system must:

• avoid to=[all subscribers]
• avoid global ACKs
• avoid any per-message work that grows with total subscribers

Status (living tracker)

Implemented (WIP, not merged)

• Local sims to stress real @peerbit/stream data plane:
  • packages/transport/pubsub/benchmark/pubsub-topic-sim.ts
  • packages/transport/pubsub/benchmark/pubsub-tree-sim.ts
• Experimental production building block: FanoutTree + FanoutChannel
  • Tree push + pull repair window
  • Stable message ids for stream-level dedup (FOUT + seq + channelKeyPrefix)
• Join/bootstrapping via bootstrap servers (tracker)
  • relays announce capacity to bootstrap nodes (TRACKER_ANNOUNCE)
  • joiners query bootstrap nodes for candidate parents (TRACKER_QUERY/TRACKER_REPLY), then dial + normal JOIN_REQ
  • Peerbit.bootstrap() also configures peer.services.fanout.setBootstraps(...)
  • Fix: avoid advertising invalid /p2p/<peerId> multiaddrs for simulated peers
• End-to-end FanoutTree sim/bench (fanout-tree-sim) with --timeoutMs + --assert* gating (FanoutTree sim/bench (end-to-end) with timeouts + CI thresholds #578)
• CI-friendly sims + nightly scale runs
  • packages/transport/pubsub/test/fanout-tree-sim.spec.ts
  • .github/workflows/nightly-sims.yml
• Real-protocol interactive sandbox + article (browser)
  • apps/peerbit-org/src/ui/FanoutProtocolSandbox.tsx
  • docs/blog/2026-02-02-interactive-fanout-visualizer.md

Next (proposed order)

• Overlay formation + maintenance as a first-class problem (metrics + scoring + re-parenting): FanoutTree overlay formation + maintenance (metrics + scoring + re-parenting) #585
• Enforce upload caps (token bucket) + overload policy + re-parent triggers: FanoutTree: enforce upload caps (token bucket) + overload policies + re-parent triggers #579
• Bounded repair improvements (neighbor-assisted / lazy summaries): FanoutTree: bounded repair improvements (neighbor-assisted / lazy summaries) #580
• Tracker hardening (limits, scoring, rate limiting) and decide if tracker should be a dedicated tiny service: FanoutTree bootstrap tracker hardening (limits, scoring, rate limiting) #581
• Relay monetization groundwork (policy semantics + metering hooks): Relay monetization groundwork (FanoutTree/DirectStream) #583

Requirements

Functional

• 1 publisher can publish at ~30 msg/s (configurable), and subscribers receive messages with bounded latency.
• Delivery works without global knowledge of subscribers.
• Nodes can configure upload limits and will not exceed them (best-effort within simulation and later production).
• Nodes can express relay preferences (e.g. “only relay if compensated / bid-based selection”).

Reliability

• Define explicit delivery goals per workload, e.g.:
  • “live”: > 99% delivered within deadline under mild churn/loss
  • “reliable”: > 99.9% delivered eventually with bounded overhead
• Repair must be bounded and local (neighbors/parent only).

Economics / incentives

• Simulate “relay earnings” based on forwarded bytes.
• Define a future-proof interface for bids/quotes (even if settlement is out-of-scope initially).

Engineering

• Provide a deterministic, local simulation harness to test 1k–10k nodes on one machine:
  • measure delivery ratio, p50/p95/p99 latency, bandwidth overhead, queue/backpressure, and “earnings” distribution.
• Add CI-friendly “small sim” tests that assert invariant thresholds.

Proposed approach (high-level)

Adopt a Plumtree-inspired architecture:

• Tree push as the steady-state data plane (economical bandwidth).
• Local pull repair (and/or gossip summaries) as the reliability layer.
• Capacity-aware admission control: each relay accepts children within an upload budget and may prefer higher bids.

Acceptance criteria (simulation)

For a configurable workload (e.g. 2k nodes, 30 msg/s, 10s, 1KB):

• Connected subscribers ≥ 99% (or defined threshold).
• Delivered ≥ 99% (or defined threshold).
• Overhead factor ≤ X (defined per reliability mode).
• No node exceeds upload cap by more than Y% (or best-effort with explicit backpressure/drop policy).

Repo notes

A living spec is tracked in:

• docs/scalable-fanout.md
• docs/fanout-tree-protocol.md

[marcus-pousette]

Status update (branch research/pubsub-large-network-testing)

What we built (WIP, not merged):

• Benchmark/sim harness to stress the real @peerbit/stream data plane locally:
  • packages/transport/pubsub/benchmark/pubsub-topic-sim.ts
  • packages/transport/pubsub/benchmark/pubsub-tree-sim.ts
• Experimental production building block: FanoutTree (/peerbit/fanout-tree/0.1.0) + FanoutChannel wrapper.
  • Tree push + pull repair window.
  • Stable message ids for dedup (FOUT + seq + channelKeyPrefix) so forwarded copies collapse in stream dedup.
• Join/bootstrapping via bootstrap servers (tracker):
  • Relays announce capacity + addrs to bootstrap nodes (TRACKER_ANNOUNCE).
  • Joiners query bootstrap nodes for candidate parents (TRACKER_QUERY/TRACKER_REPLY), then dial + normal JOIN_REQ.
  • Peerbit.bootstrap() now also configures peer.services.fanout.setBootstraps(...) so it “just works”.
• Docs/specs:
  • docs/scalable-fanout.md (living spec + Definition of Done)
  • docs/fanout-tree-protocol.md (wire format, incl tracker messages)

What we learned / roadmap adjustment:

• We can avoid “subscription gossip exploding” by making steady-state delivery a bounded-degree overlay (tree) and limiting any global-ish logic to bootstrapping/rendezvous.
• Bootstrapping needed to be first-class: “randomly try any connected peer” isn’t enough for large fanout; we added a tracker-style rendezvous (capacity announcements + redirect via candidate list).
• Upload caps can’t only be “maxChildren”; real enforcement likely needs per-node token bucket / backpressure policy + re-parenting triggers.

Next steps (proposed order):

• Add a dedicated FanoutTree sim/bench (real protocol end-to-end, uses tracker join), with --timeoutMs and CI-friendly small assertions.
• Implement real upload enforcement (token bucket) + overload behavior (drop/re-parent) with metrics.
• Improve repair (optionally neighbor-assisted / lazy summaries) while keeping repair bounded.
• Harden tracker behavior (limits, scoring, rate limiting) and decide whether bootstraps should run full FanoutTree or a tiny tracker-only service.

[marcus-pousette]

Update (local branch WIP)

• Added end-to-end fanout-tree-sim benchmark that exercises real FanoutTree join via bootstrap trackers + data forwarding over real @peerbit/stream.
  • Supports global --timeoutMs (defaults to 300000) and basic threshold gating (--assertMinJoinedPct, --assertMinDeliveryPct, --assertMaxUploadFracPct).
• Implemented bootstrap-tracker messages in the FanoutTree protocol (TRACKER_ANNOUNCE/TRACKER_QUERY/TRACKER_REPLY) so join works without global membership.
• Fixed an announce edge-case in sims: simulated peer ids (e.g. sim-0) aren’t valid /p2p/<peerId> values, so we now fall back to advertising raw multiaddrs if /p2p/... encoding fails.

Example sanity run:

• pnpm -C packages/transport/pubsub run bench -- fanout-tree-sim --nodes 3 --bootstraps 1 --subscribers 1 --messages 5 --msgRate 10 --msgSize 32 --timeoutMs 300000 --assertMinJoinedPct 100 --assertMinDeliveryPct 100

Next:

• Wire a small sim run into CI (extend FanoutTree sim/bench (end-to-end) with timeouts + CI thresholds #578) + start enforcing upload caps with a token bucket + overload policy (FanoutTree: enforce upload caps (token bucket) + overload policies + re-parent triggers #579).

[marcus-pousette]

Quick status (2026-01-31):

• FanoutTree sim now supports global timeouts + data loss + churn (offline peers) knobs.
• Loss injection was refined to be priority-aware so control-plane can keep progressing while we drop payload frames.
• FanoutTree repair now retries periodically (not only when new data arrives).

Next: start tightening CI thresholds on small churn+loss workloads, then iterate on bounded repair + overload behavior until we hit the “Definition of Done” targets.

[marcus-pousette]

Created sub-issue for incentives/monetization groundwork: #583 (Relay monetization groundwork).\n\nThis focuses on clarifying bid/ask semantics + adding lightweight metering/accounting hooks in @peerbit/stream and per-child accounting in FanoutTree, while keeping settlement out-of-scope for now.

[marcus-pousette]

Refreshed the parent issue status based on PR #582 progress (CI sims + nightly runs + interactive sandbox).

New sub-issue: #585 — treats overlay formation (metrics + parent scoring + re-parenting hysteresis) as a first-class problem so we can optimize/judge convergence without guessing.