Skip to content

Commit 9c1f361

Browse files
auscaster0state OSS
andauthored
feat(vision): add realtime intelligence pipeline
* docs: refresh distribution blowup status * build: gate ppa publishing behind repo variable * chore: bump sourcey to 3.3.10, fixes doxygen index links and node 24 compat * build: default ppa target to 0state * build: add temporary launchpad signing workflow * fix: invoke ppa publish script with bash * chore: remove temporary launchpad signing workflow * fix: target noble for ppa publishing * build: validate alpine and fedora packaging * fix: accept shared llhttp targets for system builds * build: source conan recipe from tagged releases * build: add conan release pinning flow * release: prepare 2.4.1 * fix: upstream package-manager compatibility followups * release: prepare 2.4.2 * build: finalize 2.4.2 package metadata * debian: force release build type for ppa builds * debian: ignore vendored dependency artifacts in ppa builds * debian: reuse accepted orig tarball for revision uploads * debian: build revision uploads from accepted orig source * build: refresh rpm changelog for 2.4.2 * packaging: prepare MacPorts and Spack recipes * bump sourcey to 3.4.2 * bump sourcey to 3.4.1 * packaging: add package helper scripts * add icey logo to docs * packaging: refresh Homebrew and MacPorts deps * packaging: sync conda-forge recipe * packaging: add openSUSE seeds * base: export shared-library symbols on Unix * packaging: backport Unix export fix to conda-forge recipe * distribution: sync package recipes and Unix exports * av: support older CoreAudio SDKs * ci: fix packaging and CoreAudio regressions * docs: refresh icey docs and branding * deps: bump sourcey to 3.4.4 * build: publish icey image on release * build: fix downstream packaging compatibility * release: prepare 2.4.3 * build: pin release archives for 2.4.3 * ci: dispatch finalized release publishing * add webrtc receive jitter buffer * update docs, add graft/speech/vision modules, bump sourcey * docs: update changelog for jitter buffer, graft/speech/vision modules * feat(vision): add realtime intelligence pipeline --------- Co-authored-by: 0state OSS <oss@0state.com>
1 parent 4111d97 commit 9c1f361

18 files changed

Lines changed: 1323 additions & 92 deletions

docs/concepts/packetstream.md

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,51 @@ Use this when:
216216

217217
This is the clean way to move work, not an excuse to make the whole graph vaguely asynchronous.
218218

219+
## Branch Topology
220+
221+
The real-time intelligence shape is not one long linear stream.
222+
223+
It is:
224+
225+
```text
226+
decoded ingress
227+
|-> delivery branch -> encode/send -> browser
228+
|-> detect branch -> sample -> async clone boundary -> detector -> events
229+
```
230+
231+
That means the two branches share one decoded source, but they do not share one latency budget.
232+
233+
- the delivery branch stays close to the current transport hot path
234+
- the detect branch is allowed to sample, queue, and drop stale work
235+
- the first explicit `Cloned` or `Retained` adapter on the detect branch is the ownership handoff
236+
237+
In practice the detect branch should cross that boundary before any worker-thread or slow detector stage:
238+
239+
```text
240+
ingress PacketStream
241+
|-> delivery PacketStream
242+
|-> detect PacketStream -> AsyncPacketQueue -> detector
243+
```
244+
245+
That keeps the contract honest:
246+
247+
- decode once
248+
- branch from decoded packets, not from encoded sender output
249+
- keep browser delivery independent from detector backlog
250+
- make the async boundary visible in the graph instead of implicit in some callback
251+
252+
### Shutdown Order
253+
254+
The safe teardown order for that topology is:
255+
256+
1. stop the detect branch
257+
2. stop the delivery branch
258+
3. stop the shared source
259+
260+
Downstream branches can stop independently. The shared source should be the last thing to stop.
261+
262+
Late packets after shutdown are expected around queue boundaries. They should be dropped cleanly after close, not dispatched into already-closed downstream sinks.
263+
219264
## Real Patterns
220265

221266
### Webcam to browser
@@ -277,4 +322,3 @@ That is what gives the library a coherent core instead of five separate async mo
277322
- [Runtime Contracts](runtime-contracts.md) for the loop, signal, and ownership rules around the pipeline
278323
- [WebRTC](../modules/webrtc.md) for the media send and receive layers built on top of it
279324
- [AV](../modules/av.md) for capture, decode, encode, and mux components
280-

src/base/tests/basetests.cpp

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1028,6 +1028,9 @@ int main(int argc, char** argv)
10281028
describe("packet stream async queue clone boundary", new PacketStreamAsyncQueueCloneBoundaryTest);
10291029
describe("packet stream retention contract", new PacketStreamRetentionContractTest);
10301030
describe("packet stream shared source branch clone boundary", new PacketStreamSharedSourceBranchCloneBoundaryTest);
1031+
describe("packet stream branch fanout sequence", new PacketStreamBranchFanoutSequenceTest);
1032+
describe("packet stream branch teardown order", new PacketStreamBranchTeardownOrderTest);
1033+
describe("packet stream async late drop after close", new PacketStreamAsyncLateDropAfterCloseTest);
10311034
// describe("multi packet stream", new MultiPacketStreamTest);
10321035

10331036
test::runAll();

src/base/tests/basetests.h

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,10 @@
3636
#include "icy/util.h"
3737

3838
#include <cstring>
39+
#include <mutex>
3940
#include <sstream>
41+
#include <thread>
42+
#include <vector>
4043

4144

4245
using icy::test::Test;
@@ -476,6 +479,89 @@ struct MockBurstPacketSource : public PacketSource
476479
}
477480
};
478481

482+
483+
struct BranchLifecyclePacketSource : public PacketSource
484+
, public basic::Startable
485+
{
486+
PacketSignal emitter;
487+
Thread runner;
488+
std::atomic<bool> running{false};
489+
std::atomic<bool> started{false};
490+
std::atomic<int> sent{0};
491+
std::vector<std::string>* lifecycle = nullptr;
492+
std::mutex* lifecycleMutex = nullptr;
493+
494+
BranchLifecyclePacketSource(std::vector<std::string>& lifecycleLog,
495+
std::mutex& lifecycleLogMutex)
496+
: PacketSource(emitter)
497+
, lifecycle(&lifecycleLog)
498+
, lifecycleMutex(&lifecycleLogMutex)
499+
{
500+
}
501+
502+
~BranchLifecyclePacketSource() override
503+
{
504+
stop();
505+
}
506+
507+
void start() override
508+
{
509+
if (running.exchange(true))
510+
return;
511+
512+
started = true;
513+
record("source:start");
514+
515+
runner.start([this]() {
516+
while (running.load()) {
517+
const std::string payload = "pkt" + std::to_string(sent.fetch_add(1));
518+
RawPacket packet(payload.data(), payload.size());
519+
emitter.emit(packet);
520+
icy::sleep(2);
521+
}
522+
});
523+
}
524+
525+
void stop() override
526+
{
527+
const bool wasRunning = running.exchange(false);
528+
if (wasRunning)
529+
record("source:stop");
530+
531+
if (started.exchange(false) && Thread::currentID() != runner.tid())
532+
runner.join();
533+
}
534+
535+
void record(const std::string& entry)
536+
{
537+
std::lock_guard<std::mutex> guard(*lifecycleMutex);
538+
lifecycle->push_back(entry);
539+
}
540+
};
541+
542+
543+
class GatedAsyncPacketQueue : public AsyncPacketQueue<>
544+
{
545+
public:
546+
explicit GatedAsyncPacketQueue(int maxSize = 16)
547+
: AsyncPacketQueue<>(maxSize)
548+
{
549+
}
550+
551+
std::atomic<bool> enteredDispatch{false};
552+
std::atomic<bool> releaseDispatch{false};
553+
554+
protected:
555+
void dispatch(IPacket& packet) override
556+
{
557+
enteredDispatch = true;
558+
while (!releaseDispatch.load())
559+
icy::sleep(1);
560+
561+
AsyncPacketQueue<>::dispatch(packet);
562+
}
563+
};
564+
479565
class PacketStreamTest : public Test
480566
{
481567
int numPackets;
@@ -1191,6 +1277,199 @@ class PacketStreamSharedSourceBranchCloneBoundaryTest : public Test
11911277
};
11921278

11931279

1280+
class PacketStreamBranchFanoutSequenceTest : public Test
1281+
{
1282+
std::mutex receivedMutex;
1283+
std::vector<std::string> deliveryReceived;
1284+
std::vector<std::string> detectReceived;
1285+
1286+
static std::string packetString(IPacket& packet)
1287+
{
1288+
return std::string(packet.data(), packet.size());
1289+
}
1290+
1291+
void run()
1292+
{
1293+
PacketSignal ingress;
1294+
PacketStream deliveryBranch("delivery-branch");
1295+
PacketStream detectBranch("detect-branch");
1296+
auto detectQueue = std::make_shared<AsyncPacketQueue<>>(16);
1297+
const std::vector<std::string> expected = {"frame-0", "frame-1", "frame-2", "frame-3"};
1298+
1299+
deliveryBranch.attachSource(ingress);
1300+
detectBranch.attachSource(ingress);
1301+
detectBranch.attach(detectQueue, 0);
1302+
1303+
deliveryBranch.emitter += [&](IPacket& packet) {
1304+
std::lock_guard<std::mutex> guard(receivedMutex);
1305+
deliveryReceived.push_back(packetString(packet));
1306+
};
1307+
detectBranch.emitter += [&](IPacket& packet) {
1308+
std::lock_guard<std::mutex> guard(receivedMutex);
1309+
detectReceived.push_back(packetString(packet));
1310+
};
1311+
1312+
deliveryBranch.start();
1313+
detectBranch.start();
1314+
1315+
for (const auto& payload : expected) {
1316+
RawPacket packet(payload.data(), payload.size());
1317+
ingress.emit(packet);
1318+
}
1319+
1320+
expect(test::waitFor([&]() {
1321+
std::lock_guard<std::mutex> guard(receivedMutex);
1322+
return detectReceived.size() == expected.size();
1323+
}, 2000));
1324+
1325+
deliveryBranch.close();
1326+
detectBranch.close();
1327+
1328+
std::lock_guard<std::mutex> guard(receivedMutex);
1329+
expect(deliveryReceived == expected);
1330+
expect(detectReceived == expected);
1331+
expect(detectQueue->retention() == PacketRetention::Cloned);
1332+
}
1333+
};
1334+
1335+
1336+
class PacketStreamBranchTeardownOrderTest : public Test
1337+
{
1338+
std::mutex lifecycleMutex;
1339+
std::vector<std::string> lifecycle;
1340+
std::atomic<int> deliveryReceived{0};
1341+
std::atomic<int> detectReceived{0};
1342+
1343+
void record(const std::string& entry)
1344+
{
1345+
std::lock_guard<std::mutex> guard(lifecycleMutex);
1346+
lifecycle.push_back(entry);
1347+
}
1348+
1349+
int indexOf(const std::string& entry)
1350+
{
1351+
std::lock_guard<std::mutex> guard(lifecycleMutex);
1352+
for (size_t index = 0; index < lifecycle.size(); ++index) {
1353+
if (lifecycle[index] == entry)
1354+
return static_cast<int>(index);
1355+
}
1356+
1357+
return -1;
1358+
}
1359+
1360+
void run()
1361+
{
1362+
PacketStream sourceStream("ingress-source");
1363+
auto source = std::make_shared<BranchLifecyclePacketSource>(lifecycle, lifecycleMutex);
1364+
PacketStream deliveryBranch("delivery-branch");
1365+
PacketStream detectBranch("detect-branch");
1366+
auto detectQueue = std::make_shared<AsyncPacketQueue<>>(16);
1367+
1368+
sourceStream.attachSource(source, true);
1369+
deliveryBranch.attachSource(sourceStream.emitter);
1370+
detectBranch.attachSource(sourceStream.emitter);
1371+
detectBranch.attach(detectQueue, 0);
1372+
1373+
deliveryBranch.emitter += [&](IPacket&) {
1374+
++deliveryReceived;
1375+
};
1376+
detectBranch.emitter += [&](IPacket&) {
1377+
++detectReceived;
1378+
};
1379+
1380+
deliveryBranch.StateChange += [&](void*, PacketStreamState& state, const PacketStreamState&) {
1381+
if (state.id() == PacketStreamState::Closed)
1382+
record("delivery:closed");
1383+
};
1384+
detectBranch.StateChange += [&](void*, PacketStreamState& state, const PacketStreamState&) {
1385+
if (state.id() == PacketStreamState::Closed)
1386+
record("detect:closed");
1387+
};
1388+
1389+
deliveryBranch.start();
1390+
detectBranch.start();
1391+
sourceStream.start();
1392+
1393+
expect(test::waitFor([&]() {
1394+
return deliveryReceived.load() > 0 && detectReceived.load() > 0;
1395+
}, 2000));
1396+
1397+
detectBranch.close();
1398+
expect(test::waitFor([&]() {
1399+
return detectBranch.closed();
1400+
}, 2000));
1401+
1402+
const int deliveryBefore = deliveryReceived.load();
1403+
expect(test::waitFor([&]() {
1404+
return deliveryReceived.load() > deliveryBefore;
1405+
}, 2000));
1406+
expect(source->running.load());
1407+
expect(indexOf("source:stop") == -1);
1408+
1409+
deliveryBranch.close();
1410+
expect(test::waitFor([&]() {
1411+
return deliveryBranch.closed();
1412+
}, 2000));
1413+
expect(source->running.load());
1414+
expect(indexOf("source:stop") == -1);
1415+
1416+
sourceStream.close();
1417+
expect(!source->running.load());
1418+
1419+
expect(indexOf("detect:closed") != -1);
1420+
expect(indexOf("delivery:closed") != -1);
1421+
expect(indexOf("source:stop") != -1);
1422+
expect(indexOf("detect:closed") < indexOf("source:stop"));
1423+
expect(indexOf("delivery:closed") < indexOf("source:stop"));
1424+
}
1425+
};
1426+
1427+
1428+
class PacketStreamAsyncLateDropAfterCloseTest : public Test
1429+
{
1430+
std::atomic<int> received{0};
1431+
1432+
void onPacket(IPacket&)
1433+
{
1434+
++received;
1435+
}
1436+
1437+
void run()
1438+
{
1439+
PacketStream stream("late-drop");
1440+
auto queue = std::make_shared<GatedAsyncPacketQueue>(16);
1441+
1442+
stream.attach(queue, 0);
1443+
stream.emitter += slot(this, &PacketStreamAsyncLateDropAfterCloseTest::onPacket);
1444+
stream.start();
1445+
1446+
stream.write(RawPacket("late", 4));
1447+
1448+
expect(test::waitFor([&]() {
1449+
return queue->enteredDispatch.load();
1450+
}, 2000));
1451+
1452+
std::atomic<bool> closeReturned{false};
1453+
std::thread closer([&]() {
1454+
stream.close();
1455+
closeReturned = true;
1456+
});
1457+
1458+
icy::sleep(25);
1459+
queue->releaseDispatch = true;
1460+
1461+
expect(test::waitFor([&]() {
1462+
return closeReturned.load();
1463+
}, 2000));
1464+
1465+
closer.join();
1466+
1467+
expect(received.load() == 0);
1468+
expect(stream.closed());
1469+
}
1470+
};
1471+
1472+
11941473
} // namespace icy
11951474

11961475

src/vision/README.md

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,21 @@ Low-latency video intelligence primitives for Icey.
44

55
The first landing is intentionally narrow:
66

7+
- `types.h` defines the stable frame, detection, track, and region model.
8+
- `VisionEvent` serializes directly to `json::Value` so the server layer can
9+
ship events over existing WebSocket signalling surfaces without another
10+
transport format.
11+
- `eventemitter.h` provides the narrow event signal surface detectors publish
12+
through.
713
- `FrameSampler` keeps the branch zero-copy on the synchronous hot path.
8-
- `DetectionQueue` is the explicit async clone boundary.
9-
- `MotionDetector` provides a minimal built-in video detector without adding
10-
external dependencies.
11-
- `VisionEvent` serializes directly to `json::Value` so it can flow into
12-
existing Symple/WebSocket surfaces without another transport format.
14+
- `VisionFramePacket` is the detector-side frame contract carrying stable
15+
metadata.
16+
- `FrameNormalizer` is the pre-inference seam. With default zero values it is a
17+
no-op metadata/materialization step; if configured it can also resize or
18+
normalize pixel format before inference.
19+
- `DetectionQueue` is the explicit bounded async clone boundary.
20+
- `MotionDetector` is the first built-in detector and now consumes
21+
`VisionFramePacket` instead of raw decode packets directly.
1322

1423
This module is video-only. Audio intelligence belongs in a future sibling
1524
module rather than a combined `ai` surface.

0 commit comments

Comments
 (0)