You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
feat(engine,nodes): add cross-node control messaging via param_bridge (#280)
* feat(engine,nodes): add cross-node control messaging via param_bridge
Introduces a generalizable pattern for cross-node control messaging
within pipeline graphs, enabling any node to send UpdateParams to
sibling nodes by name.
Phase 1 — Engine control channel in NodeContext:
- Add engine_control_tx: Option<mpsc::Sender<EngineControlMessage>>
field to NodeContext, wired in DynamicEngine::initialize_node()
- Add tune_sibling() convenience method for sending TuneNode messages
- Set to None in oneshot/stateless pipelines (not supported)
Phase 2 — core::param_bridge node:
- Terminal node that bridges data-plane packets to control-plane
UpdateParams messages on a configured target node
- Three mapping modes:
* Auto: smart per-packet-type (Transcription/Text → properties.text,
Custom → forward data as-is)
* Template: user-supplied JSON with {{ text }} placeholders
* Raw: forward extracted payload unchanged
- Designed for best_effort side branches to never stall main data flow
Phase 3 — Compositor word-wrap:
- Add word_wrap: bool field to TextOverlayConfig (default false)
- When true, uses transform.rect.width as wrap boundary
- Backward compatible — existing overlays unchanged
Phase 4 — Demo pipeline + Slint subtitle component:
- samples/slint/system/subtitle.slint: semi-transparent panel with
word-wrapped text and fade animation
- samples/pipelines/dynamic/video_moq_webcam_subtitles.yml: webcam PiP
with Whisper STT → param_bridge → Slint subtitle overlay
Data flow: mic → opus_decoder → resampler → whisper → [best_effort] →
param_bridge → UpdateParams → slint → compositor layer
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(nodes): skip unsupported packets in template mode, add unit tests
Fix template mode sending spurious UpdateParams with empty text when
receiving unsupported packet types (Audio, Video, Binary). Now skips
them consistently with auto and raw modes.
Add comprehensive unit tests for all param_bridge helper functions:
extract_text, auto_map, apply_template, raw_payload, and config
validation (24 tests).
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style(nodes): fix clippy lints in param_bridge
- Use let-else instead of if-let for template mode extract_text
- Move test module to end of file (items_after_test_module)
- Allow unwrap_used in test module (matches repo convention)
- Remove unused variable in test
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(nodes): address review findings for param_bridge
- tune_sibling() now returns Result<(), StreamKitError> instead of String
- Add optional debounce_ms config to coalesce rapid UpdateParams
- Make placeholder matching whitespace-insensitive ({{text}} and {{ text }})
- Document auto_map asymmetry (Slint-oriented default) in MappingMode doc
- Add extension path comment for future placeholders (language, confidence)
- Align error strings between early check and tune_sibling
- Register with StaticPins to fix schema endpoint ERROR log
- Fix sample pipeline: target_sample_rate (not sample_rate/channels),
model_path with tiny model, add debounce_ms to subtitle_bridge
- Add tests for debounce_ms config and whitespace-insensitive placeholders
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(nodes): normalize template placeholders before substitution
Fixes sequential replacement corruption when substituted text contains
the literal string '{{text}}'. Normalize '{{ text }}' → '{{text}}'
first, then replace once.
Adds regression test for this case.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(slint): rename reserved 'visible' property in subtitle.slint
'visible' is a built-in property on all Slint elements (including
Window). Declaring 'in property <bool> visible' causes a Slint
compilation error ('Cannot override property visible') that was
silently swallowed at the plugin FFI boundary, surfacing only as the
generic 'Plugin failed to create instance' message.
Rename to 'show' (consistent with lower_third.slint) and update the
sample pipeline template to match.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(nodes): emit telemetry from param_bridge for stream view visibility
Add TelemetryEmitter to param_bridge that emits 'stt.result' events
with text_preview when forwarding UpdateParams containing text. This
surfaces transcribed text in the stream view's telemetry timeline.
Also add a core::telemetry_tap node to the subtitle sample pipeline
between whisper and param_bridge so raw STT results (with segments)
appear in telemetry too.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(pipeline): add VAD filtering and telemetry_out to subtitle pipeline
Add Silero VAD configuration to the Whisper node (vad_threshold: 0.4,
min_silence_duration_ms: 600) so silence is filtered before inference,
improving transcription responsiveness.
Replace telemetry_tap with core::telemetry_out (matching other dynamic
pipelines like voice-agent-openai and speech-translate) to surface STT
results in the stream view telemetry timeline via best_effort side
branch.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(nodes): handle control_rx Shutdown in param_bridge select loop
Without this, the engine's shutdown_node() always hits the 5-second
timeout and force-aborts the node because param_bridge never reads
control_rx. This also prevented the pending debounce flush from
executing on shutdown.
Extracts control_rx from NodeContext before the loop to avoid borrow
conflicts with recv_with_cancellation (which borrows context
immutably).
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(param_bridge): dedup identical params, decouple telemetry, add subtitle transition
- Dedup: skip UpdateParams identical to last-sent value to avoid
redundant Slint re-renders during VAD boundary refinement.
- Telemetry: extract text preview before mapping so it works regardless
of the target node's JSON shape (decouples from properties.text).
- Debounce reset: use 1-year duration instead of 24h to avoid spurious
wakeup on long-running sessions (Duration::MAX overflows Instant).
- Docs: add note about raw_payload weight with Transcription packets;
explain one-time control_rx swap overhead.
- Subtitle transition: fade-in + slide-up when text arrives, fade-out +
slide-down when cleared (driven by active = show && text != "").
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: cargo fmt
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(sample): remove show:true from subtitle template
Let show remain an independent kill switch via controls/API.
The Slint active property (show && text != "") already handles
auto-hide when there is no text to display.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(sample): text transition effect + connection_mode syntax
- Subtitle: move transition to text element (fade + slide-up), not the
background overlay. Backdrop appears/disappears instantly.
- Fix connection_mode: was silently ignored at node level; use Map
variant syntax (in: {node, mode}) so best_effort is actually applied.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(sample): switch subtitle demo from Whisper to Parakeet TDT
Parakeet TDT is ~10x faster than Whisper on CPU with competitive
accuracy. Updates the subtitle pipeline to use plugin::native::parakeet
with the INT8 model and built-in VAD.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* misc improvements
* misc improvements
* fix: address Devin Review findings (template loop, profiling, parakeet lang)
- Fix apply_template infinite loop when replacement text contains
{{ field }} patterns by advancing cursor past each substitution
- Restore --profile release-lto to profiling build/run justfile targets
- Regenerate official-plugins.json to match plugin.yml expected_size_bytes
- Read detected language from FFI result instead of hardcoding "en"
(Parakeet v3 supports 25 languages)
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: cargo fmt slint plugin
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* refactor(slint): extract helpers to reduce cognitive complexity
Move InstanceState out of slint_thread_main and extract handle_register,
handle_render, apply_config_update, and apply_resize into separate
functions/methods. Reduces cognitive complexity from 66 to well under
the clippy threshold of 50.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(samples): fix VAD model setup and remove duplicate telemetry
- Update Requires comment to include download-silero-vad and
download-tenvad-models alongside download-parakeet-models so
a fresh checkout can run the demo without missing VAD assets.
- Remove the stt_telemetry (core::telemetry_out) node since
param_bridge::send_params() already emits stt.result telemetry,
avoiding duplicate entries in the stream view timeline.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(assets): flush tokio file after multipart upload to prevent truncated reads
tokio::fs::File::write_all returns as soon as data is copied to an
internal buffer and a blocking write task is spawned — it does NOT wait
for the blocking write to complete. When the File is dropped without
flushing, the last write may still be in-flight. A subsequent fs::read
can then see a truncated file.
This caused flaky E2E failures in the compositor-image-overlay upload
test: the image crate's into_dimensions() would fail with 'unexpected
end of file' because it was parsing a partially-written PNG.
The plugin upload handler in server/mod.rs already had this fix; apply
the same pattern to all asset upload functions (image, audio, font) in
assets.rs and plugin_assets.rs.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* refactor(assets): consolidate duplicate upload streaming logic
Extract stream_field_to_file() helper to replace three nearly-identical
functions (write_upload_stream_to_disk, write_image_upload_to_disk,
write_font_upload_to_disk). The cleanup-on-error pattern (remove partial
file) now appears exactly once via an inner async block, instead of being
repeated 3-4 times per function.
Also fixes a missing flush() in the image upload path — the audio and
font paths had the flush but image did not, which could cause the same
truncated-read race condition that was fixed for those paths.
Signed-off-by: streamkit-devin <devin@streamer45.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style(assets): apply rustfmt formatting to stream_field_to_file
Signed-off-by: streamkit-devin <devin@streamer45.com>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Signed-off-by: streamkit-devin <devin@streamer45.com>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
0 commit comments