Commit f77825a
feat: add AAC encoder native plugin with MP4/MoQ support (#261)
* feat: add AAC encoder native plugin with MP4/MoQ support
Implement an AAC-LC encoder as a native plugin using shiguredo_fdk_aac
2025.1.1, keeping non-royalty-free codec dependencies out of the core.
Plugin (plugins/native/aac-encoder/):
- NativeProcessorNode impl: f32→i16 PCM conversion, 1024-sample framing,
configurable bitrate (default 128 kbps), content_type and metadata
preservation via BinaryWithMeta packets.
Plugin SDK C ABI (v7, backward-compatible with v6):
- New CPacketType::BinaryWithMeta variant and CBinaryPacket struct to
preserve content_type and metadata across the native plugin boundary.
- Plugin host accepts both v6 and v7 plugins.
Core types:
- Add AudioCodec::Aac variant.
MP4 muxer:
- Explicit Aac match arms in content type and sample entry builders.
- New audio_codec config field for codec override.
MoQ transport (push + peer):
- AAC in moq_accepted_media_types().
- catalog_audio_codec() / resolve_audio_codec() / parse_audio_codec_config()
helpers mirroring the video codec pattern.
- audio_codec config field on MoqPushConfig and MoqPeerConfig.
Build system:
- just build-plugin-native-aac-encoder target.
- lint-plugins / fix-plugins / build-plugins-native entries.
Sample pipelines:
- oneshot/aac_encode.yml (audio-only AAC in MP4)
- oneshot/mp4_mux_aac_h264.yml (AAC + H264 in MP4)
- dynamic/moq_aac_mixing.yml (MoQ broadcasting with mixing + gain)
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: downgrade BinaryWithMeta for v6 plugins, fix audio_codec deserialization
Address two bugs found by Devin Review:
1. BinaryWithMeta (discriminant 10) was sent to v6 plugins that only
understand discriminants 0-9, causing packet drops. Fix: store the
plugin API version in InstanceState and call
downgrade_binary_with_meta() before forwarding to v6 plugins. This
converts to plain Binary, preserving the raw bytes while dropping the
content_type/metadata that v6 cannot interpret.
2. Mp4MuxerConfig.audio_codec was typed as Option<AudioCodec>, but the
AudioCodec enum has no serde rename_all attribute, so YAML values like
'aac' (lowercase) failed deserialization. Fix: change the field to
Option<String> with a case-insensitive parse helper, consistent with
MoqPeerConfig/MoqPushConfig.
Includes regression tests for the downgrade logic.
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: accept mono input in AAC encoder, upmix to stereo
The Opus decoder outputs mono (1 channel) but the AAC encoder previously
only accepted stereo (2 channels), causing an incompatible connection
error in the graph builder.
Fix: accept both mono and stereo input on the 'in' pin. Mono samples
are duplicated to both L/R channels before encoding, since the FDK AAC
library (shiguredo_fdk_aac) hardcodes stereo output.
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: address code review feedback for AAC encoder PR
- Split wildcard Aac | _ pattern into explicit arms with tracing::warn
for unrecognised future audio codecs (Critical #1)
- Parameterize DEFAULT_AUDIO_FRAME_DURATION_US by codec: Opus 20ms,
AAC ~21.333ms via const fn helpers (Suggestion #2)
- Compute AAC timestamps from frame count to avoid truncation drift:
sequence * 1024 * 1_000_000 / 48_000 (Suggestion #3)
- Document Binary vs EncodedAudio semantic mismatch in AAC encoder
output pin (Suggestion #4)
- Bundle video/audio codec into MediaCodecConfig struct for
handle_pin_management (Suggestion #5)
- Deduplicate parse_audio_codec_config: mp4.rs delegates to shared
implementation in moq/constants.rs (Nit #1)
- Document 960→1024 mixer/encoder frame size interaction and rewrite
moq_aac_mixing.yml as documented placeholder (Nit #2)
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: inline parse_audio_codec_config in mp4.rs to avoid moq feature dependency
The mp4 feature does not depend on moq, so importing from
transport::moq::constants would break --features mp4 builds.
Inline the trivial parsing logic directly in mp4.rs instead.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: use map_or for parse_mp4_audio_codec_config (clippy)
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: address second round of code review feedback
- Add TODO comments for MoqPullNode Opus hardcoding (blocked by
Binary→EncodedAudio C ABI gap)
- Add video_codec config field to MP4 muxer for accurate pre-connection
MIME hint (mirrors existing audio_codec field)
- Add make_dynamic_output_pin AAC test verifying AudioCodec::Aac is
threaded through to audio output pins
- Rewrite moq_aac_mixing.yml as runnable pipeline with MP4 muxer sink
instead of broken placeholder
- Set video_codec: h264 in mp4_mux_aac_h264.yml for correct hint
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: accept Binary in MP4 muxer input pins + fix mixed-source oneshot pipelines
- Add PacketType::Binary to MP4 muxer accepted input types so native
plugins (which output Binary via C ABI) can connect directly.
- Fix oneshot engine to detect generator root nodes (e.g. colorbars)
even when http_input nodes are present, enabling mixed-source
pipelines like AAC+H264 MP4 mux.
- Fix mp4_mux_aac_h264.yml: use explicit pin mapping (in/in_1) and
add num_inputs: 2 for dual-stream muxing.
- Fix clippy single_option_map lint on parse_mp4_video_codec_config.
Validated end-to-end:
- aac_encode.yml: AAC-LC 48kHz stereo 128kbps in MP4 container
- mp4_mux_aac_h264.yml: H.264 640x480 + AAC-LC stereo in MP4
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: use fragmented MP4 for browser MSE playback + drift-free duration_us
- Change aac_encode.yml from mode: file to mode: stream so the MP4
output contains mvex/moof atoms required by Media Source Extensions.
- Compute duration_us from frame count (next_timestamp - this_timestamp)
instead of using the truncated AAC_FRAME_DURATION_US constant, making
duration consistent with the drift-free timestamp computation.
- Remove unused AAC_FRAME_DURATION_US constant.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: MSE playback issues for AAC+H264 pipeline + MoQ YAML syntax
- Fix MSE codec string mismatch: OpenH264 at 640x480 outputs Level 3.0
(avc1.42c01e), not Level 3.1 (avc1.42c01f). MSE is strict about this
match and rejects the init segment when codecs don't match.
- Fix moq_aac_mixing.yml: use dot syntax (moq_peer.audio/data) instead
of bracket syntax (moq_peer[audio/data]) for dynamic pin references.
- Improve classify_packet docstring to document all handled packet types.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: MSE playback + MoQ AAC pipeline issues
- Initialise video_codec from config (matching audio_codec pattern)
so the muxer uses H264 even when type resolution is unavailable.
Previously video_codec was hardcoded to Av1, causing the init
segment to contain an av01 track instead of avc1.
- Fix placeholder AVC1 sample entry profile_compatibility (0 → 0xC0)
to match the SPS constraint flags in the placeholder NAL unit.
- Fix moq_aac_mixing.yml: replace unsupported bracket syntax
(mic_gain[in_0]) with simple array syntax so Needs::Multiple
auto-generates in_0/in_1 by index.
- Add codec detection tracing for easier debugging.
- Add regression tests for all three fixes.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(mp4): defer first fMP4 flush when inputs still open in skip-classification mode
In skip-classification mode (dual-input with explicit dimensions), the
safety cap on FMP4_FIRST_FLUSH_DEFER_CAP could force-flush an init
segment before all expected tracks had produced data. When the audio
path processes data much faster than the video path (e.g. file-based
audio vs. a video generator that needs font initialization), the cap
would trigger an audio-only init segment missing the expected h264
track, causing Chrome MSE to reject it with:
'Initialization segment misses expected h264 track'
The fix checks whether input channels are still open before applying the
safety cap. As long as channels remain open, a slow-starting track may
still produce data, so the flush is deferred. Once all channels close,
the cap fires normally to handle genuinely misconfigured pipelines.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(plugin-sdk): add EncodedAudio discriminant to native plugin C ABI
Add CPacketType::EncodedAudio (= 11) to the native plugin C ABI,
allowing plugins to declare EncodedAudio output types (e.g. AAC)
that are compatible with MoQ transport nodes.
The codec name is carried in the existing custom_type_id pointer
field (e.g. "aac", "opus") to preserve CPacketTypeInfo struct
layout and maintain ABI compatibility with v6/v7 plugins.
Also:
- Bump NATIVE_PLUGIN_API_VERSION to 8
- Update AAC encoder plugin to declare EncodedAudio(Aac) output
- Add CAudioCodec enum for documentation/future use
- Add secondary hard cap (FMP4_SKIP_CLASS_HARD_CAP = 30000) for
skip-classification fMP4 flush deferral to prevent unbounded
memory growth from pathological misconfiguration
- Create moq_aac_echo.yml sample pipeline for AAC echo over MoQ
- Remove outdated MoQ AAC limitation comment from moq_aac_mixing.yml
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: rustfmt formatting for EncodedAudio conversions
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(moq-peer): add subscriber_audio_codec config for transcoding pipelines
Add a new subscriber_audio_codec parameter to MoqPeerConfig that
controls the subscriber-side MoQ catalog codec independently from
the publisher output pin type (audio_codec).
This enables transcoding pipelines where the publisher sends one
codec (e.g. Opus) but the pipeline re-encodes to another (e.g. AAC)
before feeding it back to subscribers. Without this separation,
audio_codec controlled both the output pin type AND the catalog
codec, causing type mismatches in the graph builder.
Also fixes moq_aac_mixing.yml:
- Replace non-existent path/audio_only fields with correct
gateway_path/input_broadcasts/output_broadcast/allow_reconnect
- Remove dead-end mp4_muxer node; feed AAC directly back to
moq_peer for MoQ streaming
- Add client section for browser WebTransport connection
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: rustfmt formatting for subscriber_audio_codec
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(moq-peer): use publisher codec for dynamic output pins
Dynamic output pins carry data FROM the publisher, so they must use
the publisher's audio_codec — not subscriber_audio_codec. Without
this fix, non-primary broadcast output pins (created at runtime via
handle_pin_management) would be incorrectly typed with the subscriber
codec in transcoding pipelines.
Also fixes misleading FMP4_SKIP_CLASS_HARD_CAP comment: 30,000
samples ≈ 10 minutes of audio at typical AAC rates, not seconds.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(plugin-sdk): use EncodedAudio discriminant in macro metadata generation
The native_plugin_entry! and native_source_plugin_entry! macros were
using CPacketType::Binary as the fallback for non-Opus EncodedAudio
variants (e.g. AAC). This caused the host to read the plugin's output
pin type as Binary instead of EncodedAudio(Aac), even when the plugin
source correctly declares EncodedAudio(Aac).
Fix all four occurrences (processor + source macros × input + output
pins):
- type_discriminant: Binary → EncodedAudio
- custom_type_id: also populate codec name for EncodedAudio (was only
set for Custom types), so the host can round-trip the codec through
the C ABI
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* style: rustfmt formatting for plugin-sdk macro
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(moq-peer): advertise stereo channel_count for AAC in subscriber catalog
The AAC-LC encoder always outputs stereo (upmixing mono input), but
the subscriber catalog hardcoded channel_count=1. The client's
AudioRingBuffer was initialized with 1 channel from the catalog,
then received 2-channel decoded AAC frames, causing 'wrong number
of channels' errors.
Derive channel_count from the subscriber audio codec: AAC→2, Opus→1.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix: address review findings — stale comments, dead code, cap, plugin.yml
1. Update stale API version comments in plugin-native (lib.rs,
wrapper.rs) to reflect v6/v7/v8 compatibility and document that
EncodedAudio is metadata-only (no runtime packet downgrade needed).
2. Remove unused CAudioCodec enum from types.rs — codec name is
carried as a string via custom_type_id, not via this enum.
3. Lower FMP4_SKIP_CLASS_HARD_CAP from 100× (30,000 ≈ 10 min) to
10× (3,000 ≈ 1 min) for more reasonable memory bounds.
4. Fix plugin.yml: 'stereo' → 'mono or stereo' to match actual
plugin behavior (mono input is upmixed to stereo).
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: Devin AI <devin@cognition.ai>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Signed-off-by: Devin AI <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>1 parent 50d3c82 commit f77825a
File tree
33 files changed
+2057
-126
lines changed- crates
- api/src
- core/src
- engine/src
- nodes/src
- containers
- transport/moq
- peer
- plugin-native/src
- plugins/native
- aac-encoder
- src
- kokoro
- matcha
- nllb
- piper
- pocket-tts
- sensevoice
- slint
- supertonic
- vad
- whisper
- samples/pipelines
- dynamic
- oneshot
- sdks/plugin-sdk/native/src
33 files changed
+2057
-126
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1636 | 1636 | | |
1637 | 1637 | | |
1638 | 1638 | | |
| 1639 | + | |
| 1640 | + | |
| 1641 | + | |
| 1642 | + | |
| 1643 | + | |
| 1644 | + | |
| 1645 | + | |
| 1646 | + | |
| 1647 | + | |
| 1648 | + | |
| 1649 | + | |
| 1650 | + | |
| 1651 | + | |
| 1652 | + | |
| 1653 | + | |
| 1654 | + | |
| 1655 | + | |
| 1656 | + | |
| 1657 | + | |
1639 | 1658 | | |
1640 | 1659 | | |
1641 | 1660 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
477 | 477 | | |
478 | 478 | | |
479 | 479 | | |
480 | | - | |
481 | | - | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
482 | 484 | | |
483 | 485 | | |
484 | | - | |
485 | | - | |
| 486 | + | |
486 | 487 | | |
487 | 488 | | |
488 | 489 | | |
489 | | - | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
490 | 495 | | |
491 | 496 | | |
492 | 497 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
16 | 25 | | |
17 | 26 | | |
18 | 27 | | |
19 | 28 | | |
20 | 29 | | |
21 | | - | |
| 30 | + | |
22 | 31 | | |
23 | 32 | | |
24 | 33 | | |
| |||
28 | 37 | | |
29 | 38 | | |
30 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
31 | 44 | | |
32 | 45 | | |
33 | 46 | | |
| |||
132 | 145 | | |
133 | 146 | | |
134 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
135 | 201 | | |
136 | 202 | | |
137 | 203 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
92 | 102 | | |
93 | 103 | | |
94 | 104 | | |
| |||
98 | 108 | | |
99 | 109 | | |
100 | 110 | | |
| 111 | + | |
101 | 112 | | |
102 | 113 | | |
103 | 114 | | |
| |||
157 | 168 | | |
158 | 169 | | |
159 | 170 | | |
| 171 | + | |
| 172 | + | |
160 | 173 | | |
161 | 174 | | |
162 | 175 | | |
| |||
313 | 326 | | |
314 | 327 | | |
315 | 328 | | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
316 | 354 | | |
317 | 355 | | |
318 | 356 | | |
| |||
327 | 365 | | |
328 | 366 | | |
329 | 367 | | |
| 368 | + | |
| 369 | + | |
330 | 370 | | |
331 | 371 | | |
332 | 372 | | |
| |||
0 commit comments