Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .claude/skills/adding-plugins/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: adding-plugins
description: Checklist for making a StreamKit plugin official and downloadable from the registry. Use when adding a new native plugin or updating an existing one for marketplace distribution.
---

For the full plugin addition checklist, see [guide.md](guide.md).
Comment on lines +7 to +12
Copy link
Copy Markdown
Contributor Author

@staging-devin-ai-integration staging-devin-ai-integration bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 YAML frontmatter compatibility with SPDX headers

The .claude/skills/*/SKILL.md files use YAML frontmatter (--- delimiters) as their first content. When adding the SPDX HTML comment header before the frontmatter, some strict markdown/YAML parsers may not recognize the frontmatter if it's not at byte 0 of the file. However, Claude Code's SKILL.md parser (and most modern tools like GitHub, Hugo, Jekyll) handle HTML comments before frontmatter without issue. If a compatibility issue arises with skills.sh tooling, the SPDX header could alternatively be placed inside the frontmatter block as a YAML comment, though the HTML comment approach is consistent with every other .md file in this repo.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground

1 change: 1 addition & 0 deletions .claude/skills/adding-plugins/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/architecture/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: architecture
description: StreamKit crate relationships, data flow, key abstractions, and UI architecture. Use when understanding how the codebase is structured, how pipelines flow from YAML to execution, or how crates depend on each other.
---

For the full architecture guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/architecture/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/common-pitfalls/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: common-pitfalls
description: Known mistakes that coding agents frequently make in the StreamKit codebase. Use before starting any task to avoid wasted effort — covers WebSocket vs REST state, perf baselines, Bun-only tooling, SPDX headers, lint suppressions, and React.memo barriers.
---

For the full pitfalls guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/common-pitfalls/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/e2e-testing/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: e2e-testing
description: Running StreamKit E2E tests with Playwright. Use when writing, running, or debugging end-to-end tests, or when dealing with headless browser issues like IntersectionObserver visibility.
---

For the full E2E testing guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/e2e-testing/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/render-performance/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: render-performance
description: StreamKit render performance profiling infrastructure. Use when touching compositor hooks or components, optimizing render performance, or investigating cascade re-renders in React.memo'd components.
---

For the full render performance guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/render-performance/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/skills-setup/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: skills-setup
description: Recommended skills.sh packages for StreamKit development. Use when setting up curated agent skills for React, Playwright, or other tools used in the project.
---

For the full skills setup guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/skills-setup/guide.md
12 changes: 12 additions & 0 deletions .claude/skills/ui-development/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

---
name: ui-development
description: StreamKit React UI development patterns. Use when working on frontend components, state management (Jotai atoms for per-node data, Zustand for pipeline structure), or any code in ui/.
---

For the full UI development guide, see [guide.md](guide.md).
1 change: 1 addition & 0 deletions .claude/skills/ui-development/guide.md
267 changes: 82 additions & 185 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,193 +4,90 @@ SPDX-FileCopyrightText: © 2025 StreamKit Contributors
SPDX-License-Identifier: MPL-2.0
-->

# StreamKit agent notes

These notes apply to coding agents (Claude/Codex/etc.) contributing to this repo.

## Supervision requirement

Agent-assisted contributions are welcome, but should be **supervised** and **reviewed by a human** before merge. Treat agent output as untrusted: verify correctness, security, licensing, and style.

## Project basics

- **Supported platform**: Linux x86_64 (for now).
- **Primary server binary**: `skit` (crate: `streamkit-server`).
- **Dev task runner**: `just` (see `justfile`).
- **Docs**: Astro + Starlight in `docs/` (sidebar in `docs/astro.config.mjs`).
- **UI tooling**: Bun-first. Use `bun install`, `bunx` (or `bun run` scripts) for UI work—avoid npm/pnpm.

## Workflow expectations
# StreamKit Agent Notes

These notes apply to coding agents (Claude/Codex/Devin/etc.) contributing to
this repo. Agent-assisted contributions are welcome, but should be **supervised**
and **reviewed by a human** before merge.

## What is StreamKit

StreamKit is a self-hostable media processing server (Rust). A single binary
(`skit`) runs pipelines as a node graph (DAG) — via a web UI, YAML, or
WebSocket API. Two modes: **dynamic** (real-time, hot-reconfigurable) and
**oneshot** (stateless batch processing). See `agent_docs/architecture.md` for
the full architecture.

## Codebase Map

| Directory | Purpose |
|-----------|---------|
| `apps/skit/` | Server binary — HTTP/WS handlers, config, auth, plugins |
| `apps/skit-cli/` | CLI client binary (`streamkit-client`) |
| `crates/core/` | Shared traits/types — `ProcessorNode`, `Pin`, `Packet`, `NodeRegistry` |
| `crates/engine/` | Pipeline executor — graph builder, oneshot engine, dynamic actor |
| `crates/nodes/` | Built-in nodes: `audio::`, `video::`, `transport::`, `core::`, `containers::` |
| `crates/api/` | YAML pipeline parsing, WebSocket protocol, TS type generation |
| `crates/plugin-{native,wasm}/` | Plugin host adapters (FFI and WASM) |
| `sdks/plugin-sdk/` | Plugin SDKs for Rust, Go, and C |
| `ui/` | React 19 web UI (Vite + Bun) |
| `plugins/native/` | Official ML plugins (Whisper, Kokoro, NLLB, etc.) |
| `samples/` | Example pipelines (`dynamic/` and `oneshot/`), audio files, images, fonts, Slint files |
| `tests/` | Pipeline validation tests (oneshot pipeline smoke tests) |
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Codebase map references tests/ directory that doesn't exist yet

Both AGENTS.md:36 and agent_docs/architecture.md:37 document a tests/ directory for "Pipeline validation tests (oneshot pipeline smoke tests)". This directory does not exist on origin/main or in the current working tree. It was introduced in commit 70347b9 (PR #285: "feat(e2e): add headless pipeline validation tests") which is not yet merged into main. If this PR merges before #285, the documentation will reference a non-existent directory. Not flagged as a bug because it's common for doc PRs to anticipate parallel changes, and the impact is limited to documentation accuracy.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intentional — the maintainer explicitly requested adding tests/ to anticipate the parallel PR landing soon.

| `e2e/` | Playwright end-to-end tests |
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

worth mentioning we have another level of non-UI e2e tests, under tests/, which are essentially oneshot pipeline validations atm.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked for a top-level tests/ directory but it doesn't exist yet on main (only e2e/tests/). Could you clarify where these oneshot pipeline validation tests live? Happy to add the entry once I know the path — or if this is something planned, I can add it when it lands.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR is not up to date with main, you should merge.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merged main into the branch. However, tests/ still doesn't exist at the repo root after the merge — only e2e/tests/. Is it on a different branch, or something planned? Happy to add the codebase map entry once it lands.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, these are getting introduced in #279, but still makes sense to assume they are in so we don't forget later as we'll merge both soon.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying — added tests/ to both codebase maps (AGENTS.md and architecture.md) as "Pipeline validation tests (oneshot pipeline smoke tests)". Fixed in 58087fd.

| `docs/` | Astro + Starlight docs site (sidebar in `docs/astro.config.mjs`) |

## Tech Stack

- **Rust** (version pinned in `rust-toolchain.toml`), tokio, axum, wgpu
- **UI:** React 19, TypeScript, Zustand, Jotai, React Query, Radix UI, React Flow
- **Build/tooling:** `just` (task runner), Bun (UI), sccache (Rust build cache)
- **Testing:** `cargo test` (Rust), Vitest (UI), Playwright (E2E)
- **Platform:** Linux x86_64

## Workflow

- Keep PRs focused and minimal.
- Run `just test` and `just lint` when making code changes (or explain why you couldn't).
- Follow `CONTRIBUTING.md` (DCO sign-off, Conventional Commits, SPDX headers where applicable).
- **Linting discipline**: Do not blindly suppress lint warnings or errors with ignore/exception rules. Instead, consider refactoring or improving the code to address the underlying issue. If an exception is truly necessary, it **must** include a comment explaining the rationale.

## Running E2E tests

End-to-end tests live in `e2e/` and use Playwright (Chromium, headless).

1. **Build the UI** and **start the server** in one terminal:

```bash
just build-ui && SK_SERVER__MOQ_GATEWAY_URL=http://127.0.0.1:4545/moq SK_SERVER__ADDRESS=127.0.0.1:4545 just skit
```

2. **Run the tests** in a second terminal:

```bash
just e2e-external http://localhost:4545
```

### Headless-browser pitfalls

- Playwright runs headless Chromium with a default 1280×720 viewport.
Elements rendered below the fold are **not visible** to
`IntersectionObserver`. If a test relies on an element being observed
(e.g. the `<canvas>` used by the MoQ video renderer), scroll it into
view first:

```ts
const canvas = page.locator('canvas');
await canvas.scrollIntoViewIfNeeded();
```

- The `@moq/watch` `Video.Renderer` enables the `Video.Decoder` (and
therefore the `video/data` MoQ subscription) **only** when the canvas is
intersecting. Forgetting to scroll will result in a permanently black
canvas.

## Render performance profiling

StreamKit ships a two-layer profiling infrastructure for detecting render
regressions — particularly **cascade re-renders** where a slider interaction
(opacity, rotation) triggers expensive re-renders in unrelated memoized
components (`UnifiedLayerList`, `OpacityControl`, `RotationControl`, etc.).

### When to use this

- **After touching compositor hooks or components** (`useCompositorLayers`,
`CompositorNode`, or any `React.memo`'d sub-component): run the perf tests
to verify you haven't broken memoization barriers.
- **When optimising render performance**: use the baseline comparison to
measure before/after render counts and durations.
- **In CI**: Layer 1 tests run automatically via `just perf-ui` and will fail
if render counts regress beyond the 2σ threshold stored in the baseline.

### Layer 1 — Component-level regression tests (Vitest)

Fast, deterministic tests that measure hook/component render counts in
happy-dom. No browser required.

```bash
just perf-ui # runs all *.perf.test.* files
```

Key files:

| File | Purpose |
|------|---------|
| `ui/src/test/perf/measure.ts` | `measureRenders()` (components) and `measureHookRenders()` (hooks) |
| `ui/src/test/perf/compare.ts` | Baseline read/write, 2σ comparison, report formatting |
| `ui/src/hooks/useCompositorLayers.render-perf.test.ts` | Cascade re-render regression tests |
| `perf-baselines.json` (repo root) | Baseline snapshot — committed to track regressions over time |

**Cascade detection pattern**: the render-perf tests simulate rapid slider
drags (20 ticks of opacity/rotation) and assert that total render count stays
within a budget (currently ≤ 30). If callback references become unstable
(e.g. `layers` array in deps instead of `selectedLayerKind`), React.memo
barriers break and the render count will blow past the budget, failing the
test.

### Layer 2 — Interaction-level profiling (Playwright + React.Profiler)

Real-browser profiling for dev builds. Components wrapped with
`React.Profiler` push metrics to `window.__PERF_DATA__` which Playwright
tests can read via `page.evaluate()`.

```bash
just perf-e2e # requires: just skit + just ui (dev server at :3045)
```
- Run `just test` and `just lint` before submitting (or explain why you couldn't).
- Follow `CONTRIBUTING.md`: DCO sign-off (`git commit -s`), Conventional
Commits, SPDX license headers on all new files.
- **Linting discipline**: Do not blindly suppress lint warnings with
ignore/exception rules. Refactor instead. If a suppression is truly necessary,
include a comment explaining the rationale.
- **UI tooling**: Use `bun install` / `bunx` / `bun run` — never npm or pnpm.

Key files:
## Verification Commands

| File | Purpose |
| Task | Command |
|------|---------|
| `ui/src/perf/profiler.ts` | Dev-only `PerfProfiler` wrapper + `window.__PERF_DATA__` store |
| `e2e/tests/perf-helpers.ts` | `capturePerfData()` / `resetPerfData()` Playwright utilities |
| `e2e/tests/compositor-perf.spec.ts` | E2E test: creates PiP session, drags all sliders, asserts render budget |

Use Layer 2 when you need real paint/layout timing or want to profile
interactions end-to-end with actual browser rendering.

### Ad-hoc profiling with React DevTools exports

For investigating specific performance issues (e.g. "why does dragging this
slider feel sluggish?"), capture a profile from the React DevTools Profiler
and analyze it with the included script:

1. Open React DevTools > Profiler tab, record the interaction, then **Export**
the profile as JSON.
2. Run:

```bash
just analyze-profile profiling-data.json # summary + top components
just analyze-profile profiling-data.json --cascade # re-render cascade trees
just analyze-profile profiling-data.json --why # prop-change analysis
just analyze-profile profiling-data.json --commit 10 # single commit detail
just analyze-profile profiling-data.json --filter "Compositor|VideoLayer"
```

The script (`scripts/analyze-react-profile.mjs`) parses the React DevTools
profiling JSON (format v5), maps fiber IDs to component names, and reports:

- **Top components** by total self-time across all commits
- **Cascade trees** showing the full re-render chain for the heaviest commits
- **Why analysis** identifying prop-driven re-renders (potential object-reference
instability) and cascade victims (components re-rendering only because a parent
did)
- **Per-commit detail** with every component's self/actual time and trigger reason

Use this to pinpoint memoization breaks, unstable object references, and
unnecessary re-render cascades before diving into code.

### Updating the baseline

Run `just perf-ui` — the last test in the render-perf suite writes a fresh
`perf-baselines.json` (gated behind `UPDATE_PERF_BASELINE=1`, which the
`test:perf` script sets automatically). Regular `just test-ui` runs compare
against the baseline but never overwrite it. Commit the updated baseline
alongside your changes so future runs compare against the new numbers.

## Docker notes

- Official images are built from `Dockerfile` (CPU) and `Dockerfile.gpu` (GPU-tagged) via `.github/workflows/docker.yml`.
- `/healthz` is the lightweight health endpoint (also `/health`).
- Official images do not bundle ML models or plugins; they are expected to be mounted at runtime.

## Adding an official plugin

When making a plugin official and downloadable from the registry, update all of
the following:

- Plugin source under `plugins/native/<id>/` (crate metadata + README).
- Plugin metadata in `plugins/native/<id>/plugin.yml` (id, version, entrypoint,
artifact path, models, licenses, homepage/repo).
- Generate `marketplace/official-plugins.json` with
`scripts/marketplace/generate_official_plugins.py` and commit the result.
- Build list in `scripts/marketplace/build_official_plugins.sh`.
- Build prerequisites in `.github/workflows/release.yml` if new system deps are
required to compile or package the plugin.
- Bundle/registry smoke check: run `scripts/marketplace/build_registry.py` and
`scripts/marketplace/verify_bundles.py` locally.
- Portability table in `marketplace/PORTABILITY_REVIEW.md` (NEEDED deps,
RUNPATH/RPATH, recommendation).
- Docs: add/update the plugin page under
`docs/src/content/docs/reference/plugins/` and list it in
`docs/src/content/docs/reference/plugins/index.md` if applicable.
- Runtime shared libs: if the plugin needs bundled `.so` files, ensure the
bundle includes them and the entrypoint RUNPATH uses `$ORIGIN`, and update the
portability gate in `scripts/marketplace/verify_bundles.py` as needed.
- **Human review required** before bundling any new third-party shared libraries
(licensing, security, size, and distro compatibility).
| All lints | `just lint` |
| Rust lint only | `just lint-skit` (fmt + clippy with per-crate feature flags + license check) |
Copy link
Copy Markdown
Contributor Author

@staging-devin-ai-integration staging-devin-ai-integration bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 AGENTS.md incorrectly claims just lint-skit includes a license check

The Verification Commands table states just lint-skit performs "fmt + clippy with per-crate feature flags + license check", but the actual lint-skit recipe in the justfile only runs cargo fmt and cargo clippy. The license check is performed by the separate check-license-headers recipe, which is invoked by just lint but not by just lint-skit. An agent that runs only just lint-skit (believing it covers license compliance) would miss the SPDX/REUSE check entirely.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This finding is incorrect — lint-skit does include a license check. Line 235 of the justfile runs cargo deny check licenses as part of the lint-skit recipe. The description "fmt + clippy with per-crate feature flags + license check" is accurate.

| UI lint only | `just lint-ui` (prettier + eslint + tsc) |
| All tests | `just test` |
| Rust tests | `cargo test --workspace` |
| UI tests | `just test-ui` |
| Perf regression tests | `just perf-ui` |
| E2E tests | `just e2e-external http://localhost:4545` (requires running server) |
| Unused code check | `just knip-ui` |
| Build everything | `just build` |

## Docker

- Official images: `Dockerfile` (CPU) and `Dockerfile.gpu` (GPU) via `.github/workflows/docker.yml`.
- Health endpoint: `/healthz` (also `/health`).
- Standard images do not bundle ML models or plugins — mount them at runtime.
Demo images (`Dockerfile.demo`, tagged `-demo`) include bundled models and plugins.
Comment on lines +78 to +79
Copy link
Copy Markdown
Contributor Author

@staging-devin-ai-integration staging-devin-ai-integration bot Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Legacy Dockerfile.full and Dockerfile.full-gpu still exist alongside new Dockerfile.demo

The Docker section now references only Dockerfile.demo (tagged -demo), which matches the current .github/workflows/docker.yml. However, the repo still contains Dockerfile.full and Dockerfile.full-gpu files. These appear to be legacy files not referenced by any workflow. While the new documentation is accurate about the active Docker setup, the presence of these stale Dockerfiles could confuse contributors who browse the repo root. Consider removing them or adding a note.

Staging: Open in Devin

Was this helpful? React with 👍 or 👎 to provide feedback.

Debug

Playground

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged — the documentation now accurately describes the active Docker workflow (Dockerfile, Dockerfile.gpu, Dockerfile.demo). The stale Dockerfile.full / Dockerfile.full-gpu files are a pre-existing cleanup item outside the scope of this docs PR. @streamer45 — want me to file a separate issue to track removing those?

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

streamkit:full-gpu is actually used as base for our private demo deployment so I'd leave as is.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it — Dockerfile.full-gpu is in active use for the private demo deployment. No cleanup needed then. The docs are accurate as-is for the public workflow.


## Detailed Guides

Read the relevant guide **before** starting work in that area:

| Guide | When to read |
|-------|-------------|
| [`agent_docs/architecture.md`](agent_docs/architecture.md) | Understanding crate relationships, data flow, key abstractions |
| [`agent_docs/ui-development.md`](agent_docs/ui-development.md) | Working on React UI — state management, component patterns |
| [`agent_docs/e2e-testing.md`](agent_docs/e2e-testing.md) | Running E2E tests, headless-browser pitfalls |
| [`agent_docs/render-performance.md`](agent_docs/render-performance.md) | Compositor perf profiling, render regression testing |
| [`agent_docs/adding-plugins.md`](agent_docs/adding-plugins.md) | Making a plugin official — full checklist |
| [`agent_docs/common-pitfalls.md`](agent_docs/common-pitfalls.md) | Known mistakes agents make — read this first if unsure |
| [`agent_docs/skills-setup.md`](agent_docs/skills-setup.md) | Install curated [skills.sh](https://skills.sh/) packages for React, Playwright, etc. |
7 changes: 7 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
<!--
SPDX-FileCopyrightText: © 2025 StreamKit Contributors

SPDX-License-Identifier: MPL-2.0
-->

@AGENTS.md
Loading
Loading