diff --git a/.claude/skills/adding-plugins/SKILL.md b/.claude/skills/adding-plugins/SKILL.md new file mode 100644 index 00000000..41567c69 --- /dev/null +++ b/.claude/skills/adding-plugins/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: adding-plugins +description: Checklist for making a StreamKit plugin official and downloadable from the registry. Use when adding a new native plugin or updating an existing one for marketplace distribution. +--- + +For the full plugin addition checklist, see [guide.md](guide.md). diff --git a/.claude/skills/adding-plugins/guide.md b/.claude/skills/adding-plugins/guide.md new file mode 120000 index 00000000..130612af --- /dev/null +++ b/.claude/skills/adding-plugins/guide.md @@ -0,0 +1 @@ +../../../agent_docs/adding-plugins.md \ No newline at end of file diff --git a/.claude/skills/architecture/SKILL.md b/.claude/skills/architecture/SKILL.md new file mode 100644 index 00000000..191df4be --- /dev/null +++ b/.claude/skills/architecture/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: architecture +description: StreamKit crate relationships, data flow, key abstractions, and UI architecture. Use when understanding how the codebase is structured, how pipelines flow from YAML to execution, or how crates depend on each other. +--- + +For the full architecture guide, see [guide.md](guide.md). diff --git a/.claude/skills/architecture/guide.md b/.claude/skills/architecture/guide.md new file mode 120000 index 00000000..bc71144b --- /dev/null +++ b/.claude/skills/architecture/guide.md @@ -0,0 +1 @@ +../../../agent_docs/architecture.md \ No newline at end of file diff --git a/.claude/skills/common-pitfalls/SKILL.md b/.claude/skills/common-pitfalls/SKILL.md new file mode 100644 index 00000000..df8c4fd1 --- /dev/null +++ b/.claude/skills/common-pitfalls/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: common-pitfalls +description: Known mistakes that coding agents frequently make in the StreamKit codebase. Use before starting any task to avoid wasted effort — covers WebSocket vs REST state, perf baselines, Bun-only tooling, SPDX headers, lint suppressions, and React.memo barriers. +--- + +For the full pitfalls guide, see [guide.md](guide.md). diff --git a/.claude/skills/common-pitfalls/guide.md b/.claude/skills/common-pitfalls/guide.md new file mode 120000 index 00000000..14657db2 --- /dev/null +++ b/.claude/skills/common-pitfalls/guide.md @@ -0,0 +1 @@ +../../../agent_docs/common-pitfalls.md \ No newline at end of file diff --git a/.claude/skills/e2e-testing/SKILL.md b/.claude/skills/e2e-testing/SKILL.md new file mode 100644 index 00000000..fab8f454 --- /dev/null +++ b/.claude/skills/e2e-testing/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: e2e-testing +description: Running StreamKit E2E tests with Playwright. Use when writing, running, or debugging end-to-end tests, or when dealing with headless browser issues like IntersectionObserver visibility. +--- + +For the full E2E testing guide, see [guide.md](guide.md). diff --git a/.claude/skills/e2e-testing/guide.md b/.claude/skills/e2e-testing/guide.md new file mode 120000 index 00000000..324b8696 --- /dev/null +++ b/.claude/skills/e2e-testing/guide.md @@ -0,0 +1 @@ +../../../agent_docs/e2e-testing.md \ No newline at end of file diff --git a/.claude/skills/render-performance/SKILL.md b/.claude/skills/render-performance/SKILL.md new file mode 100644 index 00000000..7903de4f --- /dev/null +++ b/.claude/skills/render-performance/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: render-performance +description: StreamKit render performance profiling infrastructure. Use when touching compositor hooks or components, optimizing render performance, or investigating cascade re-renders in React.memo'd components. +--- + +For the full render performance guide, see [guide.md](guide.md). diff --git a/.claude/skills/render-performance/guide.md b/.claude/skills/render-performance/guide.md new file mode 120000 index 00000000..58716f9e --- /dev/null +++ b/.claude/skills/render-performance/guide.md @@ -0,0 +1 @@ +../../../agent_docs/render-performance.md \ No newline at end of file diff --git a/.claude/skills/skills-setup/SKILL.md b/.claude/skills/skills-setup/SKILL.md new file mode 100644 index 00000000..4ac86d0d --- /dev/null +++ b/.claude/skills/skills-setup/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: skills-setup +description: Recommended skills.sh packages for StreamKit development. Use when setting up curated agent skills for React, Playwright, or other tools used in the project. +--- + +For the full skills setup guide, see [guide.md](guide.md). diff --git a/.claude/skills/skills-setup/guide.md b/.claude/skills/skills-setup/guide.md new file mode 120000 index 00000000..2f5adb2f --- /dev/null +++ b/.claude/skills/skills-setup/guide.md @@ -0,0 +1 @@ +../../../agent_docs/skills-setup.md \ No newline at end of file diff --git a/.claude/skills/ui-development/SKILL.md b/.claude/skills/ui-development/SKILL.md new file mode 100644 index 00000000..1be4037a --- /dev/null +++ b/.claude/skills/ui-development/SKILL.md @@ -0,0 +1,12 @@ + + +--- +name: ui-development +description: StreamKit React UI development patterns. Use when working on frontend components, state management (Jotai atoms for per-node data, Zustand for pipeline structure), or any code in ui/. +--- + +For the full UI development guide, see [guide.md](guide.md). diff --git a/.claude/skills/ui-development/guide.md b/.claude/skills/ui-development/guide.md new file mode 120000 index 00000000..51285a53 --- /dev/null +++ b/.claude/skills/ui-development/guide.md @@ -0,0 +1 @@ +../../../agent_docs/ui-development.md \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index debff3ff..ed80579b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,193 +4,90 @@ SPDX-FileCopyrightText: © 2025 StreamKit Contributors SPDX-License-Identifier: MPL-2.0 --> -# StreamKit agent notes - -These notes apply to coding agents (Claude/Codex/etc.) contributing to this repo. - -## Supervision requirement - -Agent-assisted contributions are welcome, but should be **supervised** and **reviewed by a human** before merge. Treat agent output as untrusted: verify correctness, security, licensing, and style. - -## Project basics - -- **Supported platform**: Linux x86_64 (for now). -- **Primary server binary**: `skit` (crate: `streamkit-server`). -- **Dev task runner**: `just` (see `justfile`). -- **Docs**: Astro + Starlight in `docs/` (sidebar in `docs/astro.config.mjs`). -- **UI tooling**: Bun-first. Use `bun install`, `bunx` (or `bun run` scripts) for UI work—avoid npm/pnpm. - -## Workflow expectations +# StreamKit Agent Notes + +These notes apply to coding agents (Claude/Codex/Devin/etc.) contributing to +this repo. Agent-assisted contributions are welcome, but should be **supervised** +and **reviewed by a human** before merge. + +## What is StreamKit + +StreamKit is a self-hostable media processing server (Rust). A single binary +(`skit`) runs pipelines as a node graph (DAG) — via a web UI, YAML, or +WebSocket API. Two modes: **dynamic** (real-time, hot-reconfigurable) and +**oneshot** (stateless batch processing). See `agent_docs/architecture.md` for +the full architecture. + +## Codebase Map + +| Directory | Purpose | +|-----------|---------| +| `apps/skit/` | Server binary — HTTP/WS handlers, config, auth, plugins | +| `apps/skit-cli/` | CLI client binary (`streamkit-client`) | +| `crates/core/` | Shared traits/types — `ProcessorNode`, `Pin`, `Packet`, `NodeRegistry` | +| `crates/engine/` | Pipeline executor — graph builder, oneshot engine, dynamic actor | +| `crates/nodes/` | Built-in nodes: `audio::`, `video::`, `transport::`, `core::`, `containers::` | +| `crates/api/` | YAML pipeline parsing, WebSocket protocol, TS type generation | +| `crates/plugin-{native,wasm}/` | Plugin host adapters (FFI and WASM) | +| `sdks/plugin-sdk/` | Plugin SDKs for Rust, Go, and C | +| `ui/` | React 19 web UI (Vite + Bun) | +| `plugins/native/` | Official ML plugins (Whisper, Kokoro, NLLB, etc.) | +| `samples/` | Example pipelines (`dynamic/` and `oneshot/`), audio files, images, fonts, Slint files | +| `tests/` | Pipeline validation tests (oneshot pipeline smoke tests) | +| `e2e/` | Playwright end-to-end tests | +| `docs/` | Astro + Starlight docs site (sidebar in `docs/astro.config.mjs`) | + +## Tech Stack + +- **Rust** (version pinned in `rust-toolchain.toml`), tokio, axum, wgpu +- **UI:** React 19, TypeScript, Zustand, Jotai, React Query, Radix UI, React Flow +- **Build/tooling:** `just` (task runner), Bun (UI), sccache (Rust build cache) +- **Testing:** `cargo test` (Rust), Vitest (UI), Playwright (E2E) +- **Platform:** Linux x86_64 + +## Workflow - Keep PRs focused and minimal. -- Run `just test` and `just lint` when making code changes (or explain why you couldn't). -- Follow `CONTRIBUTING.md` (DCO sign-off, Conventional Commits, SPDX headers where applicable). -- **Linting discipline**: Do not blindly suppress lint warnings or errors with ignore/exception rules. Instead, consider refactoring or improving the code to address the underlying issue. If an exception is truly necessary, it **must** include a comment explaining the rationale. - -## Running E2E tests - -End-to-end tests live in `e2e/` and use Playwright (Chromium, headless). - -1. **Build the UI** and **start the server** in one terminal: - - ```bash - just build-ui && SK_SERVER__MOQ_GATEWAY_URL=http://127.0.0.1:4545/moq SK_SERVER__ADDRESS=127.0.0.1:4545 just skit - ``` - -2. **Run the tests** in a second terminal: - - ```bash - just e2e-external http://localhost:4545 - ``` - -### Headless-browser pitfalls - -- Playwright runs headless Chromium with a default 1280×720 viewport. - Elements rendered below the fold are **not visible** to - `IntersectionObserver`. If a test relies on an element being observed - (e.g. the `` used by the MoQ video renderer), scroll it into - view first: - - ```ts - const canvas = page.locator('canvas'); - await canvas.scrollIntoViewIfNeeded(); - ``` - -- The `@moq/watch` `Video.Renderer` enables the `Video.Decoder` (and - therefore the `video/data` MoQ subscription) **only** when the canvas is - intersecting. Forgetting to scroll will result in a permanently black - canvas. - -## Render performance profiling - -StreamKit ships a two-layer profiling infrastructure for detecting render -regressions — particularly **cascade re-renders** where a slider interaction -(opacity, rotation) triggers expensive re-renders in unrelated memoized -components (`UnifiedLayerList`, `OpacityControl`, `RotationControl`, etc.). - -### When to use this - -- **After touching compositor hooks or components** (`useCompositorLayers`, - `CompositorNode`, or any `React.memo`'d sub-component): run the perf tests - to verify you haven't broken memoization barriers. -- **When optimising render performance**: use the baseline comparison to - measure before/after render counts and durations. -- **In CI**: Layer 1 tests run automatically via `just perf-ui` and will fail - if render counts regress beyond the 2σ threshold stored in the baseline. - -### Layer 1 — Component-level regression tests (Vitest) - -Fast, deterministic tests that measure hook/component render counts in -happy-dom. No browser required. - -```bash -just perf-ui # runs all *.perf.test.* files -``` - -Key files: - -| File | Purpose | -|------|---------| -| `ui/src/test/perf/measure.ts` | `measureRenders()` (components) and `measureHookRenders()` (hooks) | -| `ui/src/test/perf/compare.ts` | Baseline read/write, 2σ comparison, report formatting | -| `ui/src/hooks/useCompositorLayers.render-perf.test.ts` | Cascade re-render regression tests | -| `perf-baselines.json` (repo root) | Baseline snapshot — committed to track regressions over time | - -**Cascade detection pattern**: the render-perf tests simulate rapid slider -drags (20 ticks of opacity/rotation) and assert that total render count stays -within a budget (currently ≤ 30). If callback references become unstable -(e.g. `layers` array in deps instead of `selectedLayerKind`), React.memo -barriers break and the render count will blow past the budget, failing the -test. - -### Layer 2 — Interaction-level profiling (Playwright + React.Profiler) - -Real-browser profiling for dev builds. Components wrapped with -`React.Profiler` push metrics to `window.__PERF_DATA__` which Playwright -tests can read via `page.evaluate()`. - -```bash -just perf-e2e # requires: just skit + just ui (dev server at :3045) -``` +- Run `just test` and `just lint` before submitting (or explain why you couldn't). +- Follow `CONTRIBUTING.md`: DCO sign-off (`git commit -s`), Conventional + Commits, SPDX license headers on all new files. +- **Linting discipline**: Do not blindly suppress lint warnings with + ignore/exception rules. Refactor instead. If a suppression is truly necessary, + include a comment explaining the rationale. +- **UI tooling**: Use `bun install` / `bunx` / `bun run` — never npm or pnpm. -Key files: +## Verification Commands -| File | Purpose | +| Task | Command | |------|---------| -| `ui/src/perf/profiler.ts` | Dev-only `PerfProfiler` wrapper + `window.__PERF_DATA__` store | -| `e2e/tests/perf-helpers.ts` | `capturePerfData()` / `resetPerfData()` Playwright utilities | -| `e2e/tests/compositor-perf.spec.ts` | E2E test: creates PiP session, drags all sliders, asserts render budget | - -Use Layer 2 when you need real paint/layout timing or want to profile -interactions end-to-end with actual browser rendering. - -### Ad-hoc profiling with React DevTools exports - -For investigating specific performance issues (e.g. "why does dragging this -slider feel sluggish?"), capture a profile from the React DevTools Profiler -and analyze it with the included script: - -1. Open React DevTools > Profiler tab, record the interaction, then **Export** - the profile as JSON. -2. Run: - -```bash -just analyze-profile profiling-data.json # summary + top components -just analyze-profile profiling-data.json --cascade # re-render cascade trees -just analyze-profile profiling-data.json --why # prop-change analysis -just analyze-profile profiling-data.json --commit 10 # single commit detail -just analyze-profile profiling-data.json --filter "Compositor|VideoLayer" -``` - -The script (`scripts/analyze-react-profile.mjs`) parses the React DevTools -profiling JSON (format v5), maps fiber IDs to component names, and reports: - -- **Top components** by total self-time across all commits -- **Cascade trees** showing the full re-render chain for the heaviest commits -- **Why analysis** identifying prop-driven re-renders (potential object-reference - instability) and cascade victims (components re-rendering only because a parent - did) -- **Per-commit detail** with every component's self/actual time and trigger reason - -Use this to pinpoint memoization breaks, unstable object references, and -unnecessary re-render cascades before diving into code. - -### Updating the baseline - -Run `just perf-ui` — the last test in the render-perf suite writes a fresh -`perf-baselines.json` (gated behind `UPDATE_PERF_BASELINE=1`, which the -`test:perf` script sets automatically). Regular `just test-ui` runs compare -against the baseline but never overwrite it. Commit the updated baseline -alongside your changes so future runs compare against the new numbers. - -## Docker notes - -- Official images are built from `Dockerfile` (CPU) and `Dockerfile.gpu` (GPU-tagged) via `.github/workflows/docker.yml`. -- `/healthz` is the lightweight health endpoint (also `/health`). -- Official images do not bundle ML models or plugins; they are expected to be mounted at runtime. - -## Adding an official plugin - -When making a plugin official and downloadable from the registry, update all of -the following: - -- Plugin source under `plugins/native//` (crate metadata + README). -- Plugin metadata in `plugins/native//plugin.yml` (id, version, entrypoint, - artifact path, models, licenses, homepage/repo). -- Generate `marketplace/official-plugins.json` with - `scripts/marketplace/generate_official_plugins.py` and commit the result. -- Build list in `scripts/marketplace/build_official_plugins.sh`. -- Build prerequisites in `.github/workflows/release.yml` if new system deps are - required to compile or package the plugin. -- Bundle/registry smoke check: run `scripts/marketplace/build_registry.py` and - `scripts/marketplace/verify_bundles.py` locally. -- Portability table in `marketplace/PORTABILITY_REVIEW.md` (NEEDED deps, - RUNPATH/RPATH, recommendation). -- Docs: add/update the plugin page under - `docs/src/content/docs/reference/plugins/` and list it in - `docs/src/content/docs/reference/plugins/index.md` if applicable. -- Runtime shared libs: if the plugin needs bundled `.so` files, ensure the - bundle includes them and the entrypoint RUNPATH uses `$ORIGIN`, and update the - portability gate in `scripts/marketplace/verify_bundles.py` as needed. -- **Human review required** before bundling any new third-party shared libraries - (licensing, security, size, and distro compatibility). +| All lints | `just lint` | +| Rust lint only | `just lint-skit` (fmt + clippy with per-crate feature flags + license check) | +| UI lint only | `just lint-ui` (prettier + eslint + tsc) | +| All tests | `just test` | +| Rust tests | `cargo test --workspace` | +| UI tests | `just test-ui` | +| Perf regression tests | `just perf-ui` | +| E2E tests | `just e2e-external http://localhost:4545` (requires running server) | +| Unused code check | `just knip-ui` | +| Build everything | `just build` | + +## Docker + +- Official images: `Dockerfile` (CPU) and `Dockerfile.gpu` (GPU) via `.github/workflows/docker.yml`. +- Health endpoint: `/healthz` (also `/health`). +- Standard images do not bundle ML models or plugins — mount them at runtime. + Demo images (`Dockerfile.demo`, tagged `-demo`) include bundled models and plugins. + +## Detailed Guides + +Read the relevant guide **before** starting work in that area: + +| Guide | When to read | +|-------|-------------| +| [`agent_docs/architecture.md`](agent_docs/architecture.md) | Understanding crate relationships, data flow, key abstractions | +| [`agent_docs/ui-development.md`](agent_docs/ui-development.md) | Working on React UI — state management, component patterns | +| [`agent_docs/e2e-testing.md`](agent_docs/e2e-testing.md) | Running E2E tests, headless-browser pitfalls | +| [`agent_docs/render-performance.md`](agent_docs/render-performance.md) | Compositor perf profiling, render regression testing | +| [`agent_docs/adding-plugins.md`](agent_docs/adding-plugins.md) | Making a plugin official — full checklist | +| [`agent_docs/common-pitfalls.md`](agent_docs/common-pitfalls.md) | Known mistakes agents make — read this first if unsure | +| [`agent_docs/skills-setup.md`](agent_docs/skills-setup.md) | Install curated [skills.sh](https://skills.sh/) packages for React, Playwright, etc. | diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..7d81fd84 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,7 @@ + + +@AGENTS.md diff --git a/agent_docs/adding-plugins.md b/agent_docs/adding-plugins.md new file mode 100644 index 00000000..0a0bee6a --- /dev/null +++ b/agent_docs/adding-plugins.md @@ -0,0 +1,33 @@ + + +# Adding an Official Plugin + +When making a plugin official and downloadable from the registry, update all of +the following: + +- Plugin source under `plugins/native//` (crate metadata + README). +- Plugin metadata in `plugins/native//plugin.yml` (id, version, entrypoint, + artifact path, models, licenses, homepage/repo). +- Generate `marketplace/official-plugins.json` with + `scripts/marketplace/generate_official_plugins.py` and commit the result. +- Build list in `scripts/marketplace/build_official_plugins.sh`. +- Build prerequisites in `.github/workflows/release.yml` if new system deps are + required to compile or package the plugin. +- Bundle/registry smoke check: run `scripts/marketplace/build_registry.py` and + `scripts/marketplace/verify_bundles.py` locally. +- Portability review: run `scripts/marketplace/verify_bundles.py` which checks + NEEDED deps, RUNPATH/RPATH, and reports portability issues. +- Docs: add/update the plugin page under + `docs/src/content/docs/reference/plugins/` and list it in + `docs/src/content/docs/reference/plugins/index.md` if applicable. +- Runtime shared libs: if the plugin needs bundled `.so` files, ensure the + bundle includes them and the entrypoint RUNPATH uses `$ORIGIN`, and update the + portability gate in `scripts/marketplace/verify_bundles.py` as needed. +- **Models**: if the plugin relies on ML models, upload them to the StreamKit + Hugging Face repo so they remain accessible indefinitely (license permitting). +- **Human review required** before bundling any new third-party shared libraries + (licensing, security, size, and distro compatibility). diff --git a/agent_docs/architecture.md b/agent_docs/architecture.md new file mode 100644 index 00000000..176343ca --- /dev/null +++ b/agent_docs/architecture.md @@ -0,0 +1,116 @@ + + +# StreamKit Architecture + +## Purpose + +StreamKit is a self-hostable media processing server (written in Rust). A single +binary (`skit`) runs pipelines composed as a node graph (DAG) of built-in nodes, +plugins, and scriptable logic — via a web UI, YAML, or WebSocket API. + +Two pipeline modes: + +- **Dynamic (real-time):** Long-running, hot-reconfigurable sessions (voice + agents, live streams) managed via the web UI or WebSocket API. +- **Oneshot (stateless):** Request/response batch processing (transcription, + file conversion) via HTTP API. + +## Workspace Structure + +``` +apps/skit/ Server binary — HTTP/WS handlers, config, auth, plugin management +apps/skit-cli/ CLI client binary (streamkit-client) +crates/core/ Shared traits and types — ProcessorNode, Pin, Packet, NodeRegistry +crates/engine/ Pipeline executor — graph_builder, oneshot engine, dynamic actor +crates/nodes/ All built-in processing nodes (audio, video, transport, core, containers) +crates/api/ YAML pipeline parsing, WebSocket protocol types, TS type generation +crates/plugin-native/ Host-side FFI adapter for native (.so) plugins +crates/plugin-wasm/ Host-side WASM plugin runtime (wasmtime) +sdks/plugin-sdk/ Plugin SDK for Rust, Go, and C (native + WASM targets) +ui/ React web UI — node graph editor, compositor canvas, views +plugins/native/ Official ML plugins (Whisper, Kokoro, NLLB, SenseVoice, etc.) +samples/ Example pipelines (dynamic/ and oneshot/), audio, images, fonts, Slint files +tests/ Pipeline validation tests (oneshot pipeline smoke tests) +e2e/ Playwright end-to-end tests +docs/ Astro + Starlight documentation site (sidebar in docs/astro.config.mjs) +scripts/ Build, analysis, and marketplace tooling +``` + +## Crate Dependency Flow + +``` +server (apps/skit) → engine → nodes → core + ↓ ↓ + api ─────────────→ ↗ (api also depends on core) + ↓ + plugin-native / plugin-wasm → core +``` + +(Arrows point from dependent to dependency.) + +- **`streamkit-core`** defines the foundational abstractions: `ProcessorNode` + trait, `NodeContext`, `Packet` (Audio/Video/Binary), `Pin` system + (Input/Output with typed cardinality), `NodeRegistry`, `NodeState` lifecycle. +- **`streamkit-api`** defines the WebSocket message contract (JSON) and YAML + pipeline format compilation. +- **`streamkit-engine`** takes a compiled pipeline graph and runs it. + `graph_builder` wires nodes together. `oneshot` handles batch pipelines. + The `dynamic` feature flag enables the `DynamicEngine` actor for live + sessions with hot-reconfiguration. +- **`streamkit-nodes`** registers all built-in nodes organized by domain: + `audio::` (codecs, filters), `video::` (colorbars, compositor, encoders), + `transport::` (HTTP, MoQ, RTMP, MSE), `core::` (file I/O, script, pacer, + passthrough), `containers::` (MP4, OGG, WAV, WebM). +- **`streamkit-server`** (`apps/skit`) is the final binary — axum HTTP server, + WebSocket handlers, plugin loading, config, auth, MoQ gateway. + +## Key Abstractions + +| Concept | Description | +|---------|-------------| +| **Node** | Atomic processing unit implementing `ProcessorNode`, runs as a tokio task | +| **Pin** | Typed input/output port on a node (e.g., `audio/data`, `video/frame`) | +| **Packet** | Data unit flowing between pins — `AudioFrame`, `VideoFrame`, `Binary`, etc. | +| **PinCardinality** | Connection limits: `One`, `Broadcast`, `Dynamic` | +| **Passthrough** | Pin type that defers concrete type resolution to upstream | +| **NodeRegistry** | Factory + discovery system for all available node kinds | +| **DynamicEngine** | Actor managing a live node graph with add/remove/reconnect | +| **Oneshot** | Stateless batch pipeline — request in, result out, then teardown | + +## Pipeline Lifecycle (Data Flow) + +1. **Definition:** User writes YAML or uses the web UI to define a pipeline +2. **Compilation:** `streamkit_api::yaml` parses YAML → `Pipeline` struct with + nodes, connections, and params +3. **Graph Building:** `engine::graph_builder` resolves pin types, validates + connections, creates node instances from the registry +4. **Execution:** Each node runs as an async tokio task, connected via channels. + The engine distributes packets between pins according to the graph topology. +5. **For dynamic pipelines:** The `DynamicEngine` actor accepts live mutations + (add/remove nodes, update params, reconnect) via its `DynamicEngineHandle`. + +## UI Architecture + +The web UI (`ui/`) is a React 19 + TypeScript SPA: + +- **State management:** Two complementary layers driven by WebSocket events: + - **Jotai atoms** (`ui/src/stores/sessionAtoms.ts`) — primary store for + high-frequency per-node data (states, stats, view data, params). Per-node + atom families confine re-renders to the affected node's components. + - **Zustand** (`ui/src/stores/sessionStore.ts`) — pipeline structure and + connection management (low-frequency CRUD). Also receives node state/stats + writes for transitional compatibility (being migrated to Jotai). + - **React Query** (`@tanstack/react-query`) — REST API data (font/image/audio/ + plugin assets, session list). +- **WebSocket-driven:** The WS service (`ui/src/services/websocket.ts`) batches + high-frequency updates via `requestAnimationFrame` and writes to Jotai atoms + first, with a transitional Zustand write for consumers not yet migrated. +- **Views:** Design (node graph editor + compositor canvas), Monitor (live + metrics), Convert (oneshot pipelines), Stream (dynamic MoQ pipelines). +- **Node graph:** Built on `@xyflow/react` (React Flow). +- **Compositor:** Custom canvas component with layer management, drag-resize, + text/image overlays. diff --git a/agent_docs/common-pitfalls.md b/agent_docs/common-pitfalls.md new file mode 100644 index 00000000..1aad15c4 --- /dev/null +++ b/agent_docs/common-pitfalls.md @@ -0,0 +1,110 @@ + + +# Common Agent Pitfalls + +Known mistakes that coding agents frequently make in this codebase. Read this +before starting work to avoid wasted effort and review cycles. + +## UI State: Use WebSocket-Driven State (Jotai + Zustand) + +**Never** intermix REST fetches with WebSocket state for pipeline, node, or +runtime data. + +High-frequency per-node data (states, stats, view data, params) lives in +**Jotai atoms** (`ui/src/stores/sessionAtoms.ts`). The WebSocket service +batches updates via `requestAnimationFrame` and writes to Jotai first. A +transitional Zustand write is kept for consumers not yet migrated. + +Pipeline structure and connections are managed in the **Zustand** session store +(`ui/src/stores/sessionStore.ts`). + +REST snapshots can be stale relative to WS events (e.g., `RuntimeSchemasUpdated` +arriving before or after a REST `fetchPipeline`). If you need node state/stats +in a component, read from the Jotai atoms. For pipeline structure, read from the +Zustand session store. Never fetch either separately via REST. + +## Never Commit `perf-baselines.json` Changes + +The `perf-baselines.json` file at the repo root should **only** be committed by +human maintainers who have benchmarked on bare-metal idle systems. If `just +perf-ui` modifies the file, revert it before committing: + +```bash +git checkout -- perf-baselines.json +``` + +## UI Tooling: Bun Only + +Use `bun install`, `bunx`, and `bun run` for all UI work. **Never** use npm or +pnpm — they are not supported and will produce incompatible lockfiles. + +## SPDX License Headers Required + +All new files need SPDX license headers. The format varies by language: + +```rust +// SPDX-FileCopyrightText: © 2025 StreamKit Contributors +// +// SPDX-License-Identifier: MPL-2.0 +``` + +```typescript +// SPDX-FileCopyrightText: © 2025 StreamKit Contributors +// +// SPDX-License-Identifier: MPL-2.0 +``` + +```html + +``` + +```yaml +# SPDX-FileCopyrightText: © 2025 StreamKit Contributors +# +# SPDX-License-Identifier: MPL-2.0 +``` + +## Lint Suppressions Need Justification + +Do **not** blindly suppress lint warnings with `#[allow(...)]`, `eslint-disable`, +or similar. Instead, refactor or improve the code to address the underlying issue. +If a suppression is truly necessary, it **must** include a comment explaining the +rationale. + +## UI Must Be Built Before Compiling the Server + +The skit server embeds the web UI at compile time via RustEmbed. You must run +`just build-ui` before `cargo build` for the server crate, or compilation will +fail with a missing `ui/dist/` error. + +## Conventional Commits with DCO Sign-Off + +All commits must use Conventional Commit format and include DCO sign-off: + +```bash +git commit -s -m "feat(nodes): add MP3 decoder" +``` + +Types: `feat`, `fix`, `docs`, `refactor`, `perf`, `test`, `chore`, `ci`, `build` + +Scopes: `core`, `api`, `engine`, `nodes`, `server`, `client`, `ui`, `plugins` + +## Avoid `unwrap()` in Production Rust Code + +Use `Result` types and proper error handling. `unwrap()` is only acceptable in +tests. + +## React.memo Barriers in Compositor Components + +The compositor UI uses `React.memo` extensively to prevent cascade re-renders. +When modifying compositor hooks or components (`useCompositorLayers`, +`CompositorNode`, etc.), run `just perf-ui` afterward to verify you haven't +broken memoization barriers. See `agent_docs/render-performance.md` for details. diff --git a/agent_docs/e2e-testing.md b/agent_docs/e2e-testing.md new file mode 100644 index 00000000..56cc57cd --- /dev/null +++ b/agent_docs/e2e-testing.md @@ -0,0 +1,53 @@ + + +# Running E2E Tests + +End-to-end tests live in `e2e/` and use Playwright (Chromium, headless). + +## Prerequisites + +Before running E2E tests for the first time (or on a fresh checkout), install +dependencies and Playwright browsers: + +```bash +just install-e2e # installs e2e/ Bun dependencies +just install-playwright # installs headless Chromium +``` + +See `e2e/README.md` for full details on prerequisites and options. + +## Running Tests + +1. **Build the UI** and **start the server** in one terminal: + + ```bash + just build-ui && SK_SERVER__MOQ_GATEWAY_URL=http://127.0.0.1:4545/moq SK_SERVER__ADDRESS=127.0.0.1:4545 just skit + ``` + +2. **Run the tests** in a second terminal: + + ```bash + just e2e-external http://localhost:4545 + ``` + +## Headless-Browser Pitfalls + +- Playwright runs headless Chromium with a default 1280x720 viewport. + Elements rendered below the fold are **not visible** to + `IntersectionObserver`. If a test relies on an element being observed + (e.g. the `` used by the MoQ video renderer), scroll it into + view first: + + ```ts + const canvas = page.locator('canvas'); + await canvas.scrollIntoViewIfNeeded(); + ``` + +- The `@moq/watch` `Video.Renderer` enables the `Video.Decoder` (and + therefore the `video/data` MoQ subscription) **only** when the canvas is + intersecting. Forgetting to scroll will result in a permanently black + canvas. diff --git a/agent_docs/render-performance.md b/agent_docs/render-performance.md new file mode 100644 index 00000000..674f6ad9 --- /dev/null +++ b/agent_docs/render-performance.md @@ -0,0 +1,111 @@ + + +# Render Performance Profiling + +StreamKit ships a two-layer profiling infrastructure for detecting render +regressions — particularly **cascade re-renders** where a slider interaction +(opacity, rotation) triggers expensive re-renders in unrelated memoized +components (`UnifiedLayerList`, `OpacityControl`, `RotationControl`, etc.). + +## When to Use This + +- **After touching compositor hooks or components** (`useCompositorLayers`, + `CompositorNode`, or any `React.memo`'d sub-component): run the perf tests + to verify you haven't broken memoization barriers. +- **When optimising render performance**: use the baseline comparison to + measure before/after render counts and durations. +- **In CI**: Layer 1 tests run automatically via `just perf-ui` and will fail + if render counts regress beyond the 2σ threshold stored in the baseline. + +## Layer 1 — Component-Level Regression Tests (Vitest) + +Fast, deterministic tests that measure hook/component render counts in +happy-dom. No browser required. + +```bash +just perf-ui # runs all *.perf.test.* files +``` + +Key files: + +| File | Purpose | +|------|---------| +| `ui/src/test/perf/measure.ts` | `measureRenders()` (components) and `measureHookRenders()` (hooks) | +| `ui/src/test/perf/compare.ts` | Baseline read/write, 2σ comparison, report formatting | +| `ui/src/hooks/useCompositorLayers.render-perf.test.ts` | Cascade re-render regression tests | +| `perf-baselines.json` (repo root) | Baseline snapshot — committed to track regressions over time | + +**Cascade detection pattern**: the render-perf tests simulate rapid slider +drags (20 ticks of opacity/rotation) and assert that total render count stays +within a budget (currently ≤ 30). If callback references become unstable +(e.g. `layers` array in deps instead of `selectedLayerKind`), React.memo +barriers break and the render count will blow past the budget, failing the +test. + +## Layer 2 — Interaction-Level Profiling (Playwright + React.Profiler) + +Real-browser profiling for dev builds. Components wrapped with +`React.Profiler` push metrics to `window.__PERF_DATA__` which Playwright +tests can read via `page.evaluate()`. + +```bash +just perf-e2e # requires: just skit + just ui (dev server at :3045) +``` + +Key files: + +| File | Purpose | +|------|---------| +| `ui/src/perf/profiler.ts` | Dev-only `PerfProfiler` wrapper + `window.__PERF_DATA__` store | +| `e2e/tests/perf-helpers.ts` | `capturePerfData()` / `resetPerfData()` Playwright utilities | +| `e2e/tests/compositor-perf.spec.ts` | E2E test: creates PiP session, drags all sliders, asserts render budget | + +Use Layer 2 when you need real paint/layout timing or want to profile +interactions end-to-end with actual browser rendering. + +## Ad-Hoc Profiling with React DevTools Exports + +For investigating specific performance issues (e.g. "why does dragging this +slider feel sluggish?"), capture a profile from the React DevTools Profiler +and analyze it with the included script: + +1. Open React DevTools > Profiler tab, record the interaction, then **Export** + the profile as JSON. +2. Run: + +```bash +just analyze-profile profiling-data.json # summary + top components +just analyze-profile profiling-data.json --cascade # re-render cascade trees +just analyze-profile profiling-data.json --why # prop-change analysis +just analyze-profile profiling-data.json --commit 10 # single commit detail +just analyze-profile profiling-data.json --filter "Compositor|VideoLayer" +``` + +The script (`scripts/analyze-react-profile.mjs`) parses the React DevTools +profiling JSON (format v5), maps fiber IDs to component names, and reports: + +- **Top components** by total self-time across all commits +- **Cascade trees** showing the full re-render chain for the heaviest commits +- **Why analysis** identifying prop-driven re-renders (potential object-reference + instability) and cascade victims (components re-rendering only because a parent + did) +- **Per-commit detail** with every component's self/actual time and trigger reason + +Use this to pinpoint memoization breaks, unstable object references, and +unnecessary re-render cascades before diving into code. + +## Updating the Baseline + +Run `just perf-ui` — the last test in the render-perf suite writes a fresh +`perf-baselines.json` (gated behind `UPDATE_PERF_BASELINE=1`, which the +`test:perf` script sets automatically). Regular `just test-ui` runs compare +against the baseline but never overwrite it. + +> **Important:** Never commit `perf-baselines.json` changes from agent sessions. +> Only human maintainers who have benchmarked on bare-metal idle systems should +> commit baseline updates. If `just perf-ui` modifies the file, revert it before +> committing. diff --git a/agent_docs/skills-setup.md b/agent_docs/skills-setup.md new file mode 100644 index 00000000..068c8986 --- /dev/null +++ b/agent_docs/skills-setup.md @@ -0,0 +1,88 @@ + + +# Recommended Agent Skills (skills.sh) + +[skills.sh](https://skills.sh/) is an open ecosystem for curated agent skills — +reusable procedural knowledge that helps coding agents work more effectively +with specific tools and frameworks. + +Install skills with the `skills` CLI (`npx skills`). Skills are installed as +markdown files into your agent's configuration directory. + +## Recommended Skills for StreamKit + +### For UI Work (React, Frontend) + +StreamKit's UI uses React 19, Zustand, Jotai, and Radix UI. These skills +provide best practices for React development: + +```bash +# React patterns, hooks, and performance best practices +npx skills add vercel-labs/agent-skills --skill react-best-practices -y + +# Component composition and code organization patterns +npx skills add vercel-labs/agent-skills --skill composition-patterns -y + +# Visual design and CSS/layout guidelines +npx skills add vercel-labs/agent-skills --skill web-design-guidelines -y +``` + +### For E2E Testing (Playwright) + +StreamKit uses Playwright for end-to-end tests. This skill provides patterns +for browser automation and testing: + +```bash +# Playwright-based webapp testing patterns +npx skills add anthropics/skills --skill webapp-testing -y +``` + +### Install All Recommended Skills at Once + +```bash +npx skills add vercel-labs/agent-skills \ + --skill react-best-practices \ + --skill composition-patterns \ + --skill web-design-guidelines -y + +npx skills add anthropics/skills --skill webapp-testing -y +``` + +## How Skills Work + +- Skills are stored in GitHub repositories as `SKILL.md` files +- `npx skills add` clones the skill into your agent's config directory +- Skills are agent-agnostic — they work with Claude Code, Cursor, Codex, + Windsurf, and others +- Use `npx skills list` to see installed skills +- Use `npx skills update` to update to latest versions +- Use `npx skills find ` to discover new skills + +### Important: Install Location + +**Do not install skills into `.claude/skills/`** — that directory is +repo-maintained and contains committed skills that symlink to `agent_docs/`. +Running `npx skills add` there will mix vendored content into the tracked tree. + +Install user/third-party skills into a **personal** (non-tracked) location +instead. For Claude Code, use `~/.claude/skills/` (user-level). For other +agents, use the agent's per-user config directory. + +## Repo-Maintained Skills + +StreamKit maintains two sets of committed skills: + +- **`.claude/skills/`** — Agent Skills standard (`SKILL.md` + frontmatter). + Each skill is a thin wrapper that symlinks `guide.md` → `agent_docs/.md` + so Claude Code gets progressive disclosure with zero duplication. +- **`.agents/skills/`** — Devin-specific testing workflows (compositor UI, + stream views). Follow the existing pattern when adding new Devin skills. + +For skills that should work across multiple agents, use the `SKILL.md` format +in `.claude/skills/` with a symlink to the canonical guide in `agent_docs/`. + +Learn more: [skills.sh documentation](https://skills.sh/docs) diff --git a/agent_docs/ui-development.md b/agent_docs/ui-development.md new file mode 100644 index 00000000..7618ff19 --- /dev/null +++ b/agent_docs/ui-development.md @@ -0,0 +1,90 @@ + + +# UI Development Guide + +The web UI lives in `ui/` and is a React 19 + TypeScript SPA built with Vite +and Bun. + +## State Management + +StreamKit uses four complementary state layers: + +| Layer | Tool | Purpose | +|-------|------|---------| +| **Per-node session data** | Jotai atoms (`ui/src/stores/sessionAtoms.ts`) | High-frequency node states, stats, view data, params — per-node atom families confine re-renders to affected components | +| **Pipeline structure** | Zustand (`ui/src/stores/sessionStore.ts`) | Pipeline CRUD, connections, session lifecycle (low-frequency). Also receives node state/stats for transitional compat | +| **Atomic UI state** | Jotai (`ui/src/stores/`, various atoms) | Local UI concerns (selections, toggles, form state) | +| **Server state** | React Query (`@tanstack/react-query`) | REST API data (font/image/audio/plugin assets, session list) | + +### Critical Rule: WebSocket is Source of Truth + +The WebSocket service (`ui/src/services/websocket.ts`) batches high-frequency +updates via `requestAnimationFrame` and writes to **Jotai atoms first** +(`sessionAtoms.ts`), with a transitional Zustand write for consumers not yet +migrated. For per-node data (states, stats, params), read from the Jotai atoms. +For pipeline structure and connections, read from the Zustand session store. + +**Never** intermix REST fetches with WebSocket state for the same data. REST +snapshots can be stale relative to WS events (e.g., `RuntimeSchemasUpdated` +arriving before or after a REST `fetchPipeline`). + +## Component Patterns + +- **Functional components** with hooks — no class components. +- **React.memo** for expensive components, especially in the compositor + (see `agent_docs/render-performance.md`). +- **Radix UI** primitives for accessible UI components (dialogs, dropdowns, + sliders, tabs, tooltips). +- **Lucide React** for icons. +- **Motion** (Framer Motion) for animations. + +## Key Directories + +``` +ui/src/ +├── components/ UI components (CompositorCanvas, NodeEditor, etc.) +├── hooks/ Custom React hooks (useCompositorLayers, useWebSocket, etc.) +├── views/ Top-level route views (Design, Monitor, Convert, Stream) +├── stores/ Zustand stores and Jotai atoms +├── services/ API clients and WebSocket handler +├── nodes/ React Flow custom node components +├── panes/ Panel/pane components for the editor layout +├── types/ TypeScript type definitions (auto-generated from Rust) +├── constants/ Shared constants +├── utils/ Utility functions +├── test/ Test utilities and perf measurement helpers +└── perf/ Dev-only React.Profiler wrappers +``` + +## Type Generation + +TypeScript types are auto-generated from Rust structs using `ts-rs`. Run: + +```bash +just gen-types +``` + +This produces types in `ui/src/types/` that match the Rust API contract. Do not +manually edit generated type files. + +## Verification Commands + +```bash +bun run lint # prettier + eslint + tsc --noEmit +bun run test:run # vitest (all unit tests) +bun run knip # detect unused exports/dependencies +bun run build # production bundle +``` + +Or via the justfile from the repo root: + +```bash +just lint-ui # same as bun run lint +just test-ui # same as bun run test:run +just build-ui # production build +just perf-ui # render performance regression tests +```