feat: add native Slint UI plugin (#239)

* feat: add native Slint UI plugin

Port the Slint renderer from PR #237 into a standalone native plugin
using the NativeSourceNode trait from PR #238. This keeps slint and
slint-interpreter as plugin-only dependencies, avoiding committing
them to the core workspace.

The plugin renders .slint files to RGBA8 video frames at configurable
resolution and frame rate. All Slint operations run on a shared
dedicated thread with channel-based message passing to handle the
non-Send constraint of Slint types.

Includes:
- Plugin crate with config, shared thread, and NativeSourceNode impl
- justfile build/lint/fix/copy targets
- CI format check and clippy in lint-simple job
- Marketplace metadata (plugin.yml) and regenerated official-plugins.json
- Docs reference page and updated plugin index (9 -> 10)
- Sample .slint files (watermark, scoreboard, lower_third)
- Sample oneshot pipeline using plugin::native::slint

Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix: port sample pipelines from PR #237 with plugin node kind

Replace simplified steps-based oneshot pipeline with the full
compositor pipeline from PR #237 (colorbars + watermark overlay →
compositor → VP9 → WebM → http_output).

Add the missing dynamic pipeline (scoreboard + lower-third overlays
composited onto colorbars, streamed via MoQ).

Update docs example to match the correct node-based format.

All pipelines use plugin::native::slint instead of video::slint.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix: rename output pin to 'out' and add fontconfig dep to marketplace CI

- Rename plugin output pin from 'video' to 'out' to match the
  convention used by all built-in nodes. The pipeline compiler's
  'needs' syntax defaults from_pin to 'out', so using a different
  name broke graph wiring.

- Add libfontconfig1-dev to marketplace-build.yml system deps so
  the Slint plugin can be compiled by build_official_plugins.sh.

- Update docs to reflect the pin name change.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix: support parameterless construction for source config probe

The host probes source plugins by calling create_instance with no
params to read source_config().  The Slint plugin requires slint_file
which fails validation with default config.  Return a lightweight
probe-only instance when params is None so the host correctly
detects is_source=true and uses run_source() instead of
run_processor().

Without this fix the watermark node was treated as a processor,
crashed immediately (no input pins), and the compositor produced
output with only colorbars (no overlay).

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(plugin/slint): address review findings — scope guard, clone, naming, validation

- Add ClearWindow scope guard so CURRENT_WINDOW thread-local is cleared
  even if definition.create() or component.show() fails via early return
- Remove unnecessary data.clone() in static-UI render path — store first,
  then clone from cache (saves ~3.7 MB allocation per cache-miss render)
- Rename lib from slint_plugin to slint, matching the naming convention
  of all other native plugins (whisper, kokoro, vad, etc.)
- Add MAX_DIMENSION (7680 / 8K) upper bound on width/height to guard
  against config typos that would attempt multi-GB buffer allocations
- Reject absolute paths in validate_slint_asset_path() as defense-in-depth
- Document update_timers_and_animations() idempotency for multi-instance
  usage on the shared Slint thread

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(compositor): frame-aligned sync in oneshot mode prevents asymmetric drain

Cherry-picked from PR #238 branch (84364c6). In oneshot mode, verify
all active slots have pending frames before dequeuing any, so fast
sources aren't consumed ahead of slower ones.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(plugin-native): re-query source config from live instance for per-instance max_ticks

The load-time probe creates a default instance (null params) to detect
source plugins, so max_ticks is always 0 (infinite). Per-instance params
like frame_count: 300 were never applied to the tick loop limit.

Now run_source() re-queries get_source_config() on the live instance
after it's created with actual params, so the tick loop respects the
configured frame_count.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* style: format wrapper.rs

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(plugin-native): use if-let instead of match for clippy single_match_else

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(plugin-native): avoid begin_call() leak when get_source_config is None

Split the tuple pattern into chained and_then/map so begin_call() is
only invoked when get_source_config is Some, preventing an in_flight_calls
counter leak in the else branch.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* improve graphics

* feat(client): add declarative overlay controls in client section (#240)

* feat(client): add declarative overlay controls in client section

Add a `controls` array to `ClientSection` so pipeline authors can
declare interactive widgets (toggle, text, number, button) targeting
specific node properties.  The StreamView renders these controls
automatically when a session is active, sending `TuneNodeAsync` /
`UpdateParams` messages on interaction.

Rust changes:
- New `ControlType` enum and `ControlConfig` struct in `yaml.rs`
- Extend `ClientSection` with `controls: Option<Vec<ControlConfig>>`
- Add `name` field to `NodeInfo` for node-name validation
- Lint rules: `control-unknown-node`, `control-number-no-bounds`
- Register new types for TypeScript generation

UI changes:
- New `OverlayControls` component with toggle/text/number/button widgets
- New `buildParamUpdate()` utility for dot-notation → nested JSON
- Integrate `OverlayControls` into `StreamView`

Sample:
- Add controls to `video_moq_slint_scoreboard.yml` for scoreboard
  and lower-third properties

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(controls): stabilize debounce/throttle with ref pattern, add useTuneNode hook

Address Devin Review feedback:

1. TextControl: store onSend in a ref so the debounce closure is stable
   across re-renders. Pending timers now always call the latest callback
   instead of leaking stale intermediate values.

2. NumberControl: apply the same ref pattern for onSend in the throttle
   closure for consistency, preventing similar issues.

3. Extract useTuneNode hook from useSession — a lightweight hook that
   only provides tuneNodeConfig without subscribing to pipeline or
   connection state, avoiding unnecessary re-renders in OverlayControls.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(controls): address review feedback — dead code, guards, stale state, docs

1. Remove dead isDraggingRef in NumberControl (written but never read).

2. Guard buildParamUpdate against empty/malformed paths — filter empty
   segments, throw on zero valid segments. Add unit tests covering
   single/multi-segment, empty, dot-only, double-dot, and leading/
   trailing dot paths.

3. Fix ToggleControl stale checked on rapid double-click — use
   functional updater (setChecked(prev => ...)) with onSendRef
   pattern matching TextControl/NumberControl.

4. Update misleading 'stable' comment on makeSend to accurately
   describe that a new closure is created per render but child
   controls absorb this via onSendRef.

5. Move getWebSocketService() to module scope in useTuneNode since
   it returns a singleton — avoids a new reference on every render
   and keeps tuneNodeConfig deps minimal (only sessionId).

6. Document that ControlConfig.default is a UI-only hint — it seeds
   local widget state but is not sent to the server on mount.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* fix(controls): deep-merge partial nested updates in useTuneNode

useTuneNode now reads the current nodeParamsAtom state and deep-merges
the partial config before writing, so sibling nested properties are
preserved.  Previously, two controls targeting the same node but
different nested paths (e.g. properties.home_score and
properties.away_score) would clobber each other because writeNodeParams
does a shallow top-level merge.

Add deepMerge utility to controlProps.ts — recursively merges plain
objects, replaces arrays and primitives wholesale.  Includes 8 unit
tests covering nested merge, array replacement, type transitions,
successive merges, and immutability.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

* test(e2e): add Playwright test for overlay controls

Add overlay-controls.spec.ts exercising all four control types (toggle,
text, number, button).  The test:

1. Selects the new 'Test: Overlay Controls' sample pipeline (colorbars →
   sink, no plugins/MoQ required).
2. Creates a session and verifies the Pipeline Controls section renders
   with correct labels and group headings.
3. Exercises each control type and asserts the correct TuneNodeAsync /
   UpdateParams WebSocket payload is sent:
   - Toggle: sends { draw_time: false }
   - Text:   sends { label: "World" } after 300ms debounce
   - Slider: sends { properties: { width: 800 } } (dot-notation path)
   - Button: sends { reset: true }
4. Asserts no unexpected console errors.
5. Destroys the session and cleans up.

Also adds data-testid='overlay-controls' to the OverlayControls
component for scoped locators, and fixes a TS6 useRef() arity error.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

---------

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>

* fix(lint): include controls in mode-mismatch-oneshot check

The `controls` field is dynamic-only but was not included in the
`has_dynamic_fields` check in `lint_client_section`, so a oneshot
pipeline with a `controls` section would not trigger the
`mode-mismatch-oneshot` warning.  Add it to the check and update the
warning message and doc comment accordingly.

Adds a unit test verifying the warning fires for controls-only oneshot
pipelines.

Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>

---------

Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
Co-authored-by: staging-devin-ai-integration[bot] <166158716+staging-devin-ai-integration[bot]@users.noreply.github.com>

Author: 3 people

.github/workflows/marketplace-build.yml

@@ -42,7 +42,7 @@ jobs:
       - name: Install system dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y cmake pkg-config libclang-dev wget libopenblas-dev zstd patchelf python3-yaml python3-tomli
+          sudo apt-get install -y cmake pkg-config libclang-dev libfontconfig1-dev wget libopenblas-dev zstd patchelf python3-yaml python3-tomli
 
       - name: Install minisign
         run: |

.github/workflows/plugins.yml

@@ -55,6 +55,10 @@ jobs:
         working-directory: plugins/native/nllb
         run: cargo fmt -- --check
 
+      - name: Check formatting - Slint
+        working-directory: plugins/native/slint
+        run: cargo fmt -- --check
+
   # Lint plugins that can build without pre-installed native libraries
   lint-simple:
     name: Lint (Simple Plugins)
@@ -65,7 +69,7 @@ jobs:
       - name: Install system dependencies
         run: |
           sudo apt-get update
-          sudo apt-get install -y cmake pkg-config libclang-dev
+          sudo apt-get install -y cmake pkg-config libclang-dev libfontconfig1-dev
 
       - name: Install Rust toolchain
         uses: dtolnay/rust-toolchain@master
@@ -79,12 +83,17 @@ jobs:
         with:
           workspaces: |
             plugins/native/vad
+            plugins/native/slint
           cache-on-failure: true
 
       - name: Clippy - VAD
         working-directory: plugins/native/vad
         run: cargo clippy -- -D warnings
 
+      - name: Clippy - Slint
+        working-directory: plugins/native/slint
+        run: cargo clippy -- -D warnings
+
   # Lint Whisper plugin (builds whisper.cpp from source)
   lint-whisper:
     name: Lint (Whisper)

crates/api/src/bin/generate_ts_types.rs

@@ -78,6 +78,8 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
         format!("export {}", streamkit_api::yaml::OutputType::decl(&cfg)),
         format!("export {}", streamkit_api::yaml::FieldHint::decl(&cfg)),
         format!("export {}", streamkit_api::yaml::FieldType::decl(&cfg)),
+        format!("export {}", streamkit_api::yaml::ControlType::decl(&cfg)),
+        format!("export {}", streamkit_api::yaml::ControlConfig::decl(&cfg)),
     ];
 
     let output = declarations.join("\n\n");