Commit 70347b9
feat(e2e): add headless pipeline validation tests (#285)
* chore(registry): publish marketplace registry update (#283)
* chore(registry): publish marketplace registry update
* fix(marketplace): prevent plugin releases from becoming latest on GitHub
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>
* feat(e2e): add headless pipeline validation tests
Add a Rust-based test framework for validating oneshot pipelines against
a live skit server using ffprobe for output verification. No browser
required.
Architecture:
- datatest-stable discovers .yml files in samples/pipelines/test/
- Each .yml has a companion .toml sidecar with expected output metadata
- Tests POST the pipeline YAML to /api/v1/process, save the response,
and validate codec, resolution, container format via ffprobe
- HW codec tests (NVENC AV1, Vulkan Video H.264) are skipped gracefully
when the required node kind is not registered on the server
New files:
- tests/pipeline-validation/ Standalone Rust test crate
- samples/pipelines/test/*.yml 4 short test pipelines (30 frames)
- samples/pipelines/test/*.toml Expected output metadata sidecars
- justfile: test-pipelines recipe
Usage: just test-pipelines http://localhost:4545
just test-pipelines http://localhost:4545 vp9 # filter by name
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* refactor(e2e): restructure test pipelines to one-dir-per-test layout
Move from flat files to directory-based test layout:
samples/pipelines/test/<name>/pipeline.yml
samples/pipelines/test/<name>/expected.toml
Each test is self-contained in its own directory, making it easier to
add test-specific input media or extra config in the future. The
datatest-stable harness now matches on 'pipeline.yml' recursively.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(e2e): add SVT-AV1 test pipeline and CI integration
- Add svt_av1_colorbars test pipeline (SW codec, requires svt_av1 feature)
- Add pacer node to VP9 pipeline for consistency with other WebM pipelines
- Add pipeline-validation job to e2e.yml CI workflow — runs SW codec tests
(VP9, OpenH264, SVT-AV1) against a live skit server with ffprobe validation
- GPU-specific tests (NVENC AV1, Vulkan Video H.264) are skipped in CI
via the requires_node mechanism
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ci): fail explicitly when skit server doesn't start
Add HEALTHY flag to health check loop so the pipeline-validation CI job
fails with a clear error instead of proceeding to run tests against a
server that never became healthy.
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(e2e): add complete format coverage for pipeline validation tests
Extend the pipeline validation framework to support audio-only tests and
file upload pipelines, then add 8 new test cases covering all core
codecs, muxers, and demuxers:
Audio codec tests:
- opus_roundtrip: Opus encode/decode via Ogg container
- opus_mp4: Opus encode via MP4 container (file mode)
- flac_decode: FLAC decoder (symphonia) → Opus/Ogg
- mp3_decode: MP3 decoder (symphonia) → Opus/Ogg
- wav_decode: WAV demuxer (symphonia) → Opus/Ogg
Video codec/decoder tests:
- rav1e_colorbars: rav1e AV1 encoder → WebM
- vp9_roundtrip: VP9 encode → decode → re-encode roundtrip
- dav1d_roundtrip: SVT-AV1 → dav1d decode → SVT-AV1 re-encode
Framework changes:
- Expected struct now supports audio-only tests (audio_codec,
sample_rate, channels) and file uploads (input_file)
- run_pipeline() accepts optional input file for multipart upload
- validate_output() validates audio and/or video stream properties
- Test audio fixtures (Ogg/Opus, FLAC, MP3, WAV) in fixtures/
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* chore: add REUSE/SPDX license files for test audio fixtures
Adds CC0-1.0 license companion files for the generated test tone
audio fixtures (ogg, flac, mp3, wav) to satisfy the reuse-compliance
check.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(ci): add GPU pipeline validation job on self-hosted runner
Adds a 'Pipeline Validation (GPU)' job to the E2E workflow that runs
on the self-hosted GPU runner. This builds skit with gpu, svt_av1, and
dav1d_static features, starts the server, and runs all pipeline
validation tests.
Currently the NVENC AV1 and Vulkan Video H.264 tests will skip
gracefully since those features (nvcodec, vulkan_video) aren't on main
yet. Once PR #279 merges, adding those features to the build command
will enable full HW codec pipeline validation in CI.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ci): use alternate port for GPU pipeline validation server
The self-hosted GPU runner has a persistent skit instance on port
4545. Use port 4546 for the pipeline validation server to avoid
'Address already in use' errors.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ci): add libssl-dev for GPU pipeline validation runner
The pipeline-validation test crate depends on reqwest which pulls in
openssl-sys. The self-hosted GPU runner needs libssl-dev installed.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(test): correct multipart field name and audio channel config
- Use 'media' instead of 'file' for the multipart field name to match
the server's http_input binding convention.
- Set channels: 2 on all opus_encoder nodes since test fixtures are
stereo, fixing 'Incompatible connection' errors on the GPU runner.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(test): use mono audio fixtures to match Opus encoder pin type
The Opus encoder node's input pin is hardcoded to accept mono audio
(channels: 1). Regenerate all test fixtures as mono sine waves and
update pipeline configs and expected.toml files accordingly.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(ci): add cleanup step to kill skit on self-hosted runner
Self-hosted runners persist between runs, so background processes can
accumulate. Add an always-run cleanup step to kill the skit process
after tests complete.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(test): remove incompatible audio decode pipelines and fix GPU cleanup
Remove flac_decode, mp3_decode, and wav_decode test pipelines: the FLAC
decoder, MP3 decoder, and WAV demuxer all declare channels: 2 in their
static output pins, but the Opus encoder (the only audio encoder) only
accepts channels: 1. This static type mismatch causes pipeline validation
to reject the connection before any audio data flows.
Audio codec coverage is retained via opus_roundtrip (Ogg) and opus_mp4
(MP4) tests which exercise the full Opus encode/decode path.
Also fix GPU CI cleanup: use PID-based kill instead of pkill pattern
matching (the port number is in an env var, not the command line).
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* feat(ci): enable nvcodec + vulkan_video in GPU pipeline validation
Now that the branch is rebased on top of PR #279 (HW video codecs),
enable the nvcodec and vulkan_video features in the GPU CI build so
the nv_av1_colorbars and vulkan_video_h264_colorbars tests actually
run on the self-hosted GPU runner.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
* fix(test): correct doc comment for multipart field name
The doc comment said 'file' but the code uses 'media'.
Signed-off-by: Devin AI <devin@streamkit.dev>
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Co-Authored-By: Claudio Costa <cstcld91@gmail.com>
---------
Signed-off-by: StreamKit Devin <devin@streamkit.dev>
Signed-off-by: Devin AI <devin@streamkit.dev>
Co-authored-by: streamkit-bot <registry-bot@streamkit.dev>
Co-authored-by: StreamKit Devin <devin@streamkit.dev>
Co-authored-by: Claudio Costa <cstcld91@gmail.com>1 parent a6583e8 commit 70347b9
File tree
35 files changed
+3398
-0
lines changed- .github/workflows
- dist/registry
- plugins/parakeet/0.1.0
- docs/public/registry
- plugins/parakeet/0.1.0
- samples/pipelines/test
- dav1d_roundtrip
- fixtures
- nv_av1_colorbars
- openh264_colorbars
- opus_mp4
- opus_roundtrip
- rav1e_colorbars
- svt_av1_colorbars
- vp9_colorbars
- vp9_roundtrip
- vulkan_video_h264_colorbars
- tests/pipeline-validation
- src
- tests
35 files changed
+3398
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
15 | 187 | | |
16 | 188 | | |
17 | 189 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
97 | 97 | | |
98 | 98 | | |
99 | 99 | | |
| 100 | + | |
100 | 101 | | |
101 | 102 | | |
102 | 103 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
98 | 112 | | |
99 | 113 | | |
100 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
95 | 95 | | |
96 | 96 | | |
97 | 97 | | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
98 | 112 | | |
99 | 113 | | |
100 | 114 | | |
| |||
Lines changed: 57 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
Lines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1389 | 1389 | | |
1390 | 1390 | | |
1391 | 1391 | | |
| 1392 | + | |
| 1393 | + | |
| 1394 | + | |
| 1395 | + | |
| 1396 | + | |
| 1397 | + | |
| 1398 | + | |
| 1399 | + | |
| 1400 | + | |
| 1401 | + | |
| 1402 | + | |
| 1403 | + | |
1392 | 1404 | | |
1393 | 1405 | | |
1394 | 1406 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
0 commit comments