Shutting down the pipeline (packager->relay->playout) still gives errors

[jackjansen]

Tested with command

Here is the last bit of the output, from when we start to shutdown the pipeline from the sender side.

encoder: stopping thread
lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] Pipeline: asked to flush and exit.

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] Pipeline: waiting for completion

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] Pipeline: condition (remaining: 4) (4 modules in the pipeline)

encoder: thread stopped
lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [2] Pipeline: flushing...

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [2] Pipeline: flushed

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] Pipeline: condition (remaining: 3) (4 modules in the pipeline)

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] notify end-of-stream.

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] Pipeline: flushing...

lldash_packager: asynchronous warning: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] Computed duration is inferior or equal to zero (0). Inferring to 100

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] Segment [in memory] completed (size 8) (startsWithSAP=1)

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] Pipeline: flushed

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Continue transfer (14058 bytes) for URL: "http://127.0.0.1:9000/v_0_0x0/v_0_0x0-1.m4s"

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] Pipeline: condition (remaining: 2) (4 modules in the pipeline)

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Continue transfer (8 bytes) for URL: "http://127.0.0.1:9000/v_0_0x0/v_0_0x0-1.m4s"

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Ending transfer for URL: "http://127.0.0.1:9000/v_0_0x0/v_0_0x0-1.m4s"

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash-relay: t=2025-11-17T11:27:25.337 event=resource_chunk_received url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=14058
lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Processes segment (total processed: 16.000000s)
lldash-relay: t=2025-11-17T11:27:25.337 
event=resource_chunk_received url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=8
lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] notify end-of-stream.

lldash-relay: t=2025-11-17T11:27:25.337 event=chunk_sent url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=14058
lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Pipeline: flushing...
lldash-relay: t=2025-11-17T11:27:25.337 
event=resource_created url=/v_0_0x0/v_0_0x0-1.m4s
lldash-relay: t=2025-11-17T11:27:25.337 event=chunk_sent url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=8
lldash-relay: t=2025-11-17T11:27:25.337 event=request_completed method=GET url=/v_0_0x0/v_0_0x0-1.m4s status=200
lldash-relay: t=2025-11-17T11:27:25.337 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:27:25.337 event=request_completed method=PUT url=/v_0_0x0/v_0_0x0-1.m4s status=200
lldash-relay: t=2025-11-17T11:27:25.337 lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Manifest was not rewritten for on-demand and all files are being deleted.
event=connection_closed reason=client_closed

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Pipeline: flushed

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] Pipeline: condition (remaining: 1) (4 modules in the pipeline)

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] Output: metadata not equal but comparable by value. Updating.

lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-2.m4s'

lldash-relay: t=2025-11-17T11:27:25.338 event=request_received method=GET url=/v_0_0x0/v_0_0x0-2.m4s version=HTTP/1.1
lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Starting transfer to URL: "http://127.0.0.1:9000/v_0_0x0/v_0_0x0-2.m4s"

lldash_play: read: lldplay_grab_frame(handle, 0, ptr, 13926, None)
netdecoder: decoded pointcloud with 14743 points, qlen=1
lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Continue transfer (8 bytes) for URL: "http://127.0.0.1:9000/v_0_0x0/v_0_0x0-2.m4s"

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Starting transfer to URL: "http://127.0.0.1:9000/lldash_testlatency.mpd"

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Continue transfer (989 bytes) for URL: "http://127.0.0.1:9000/lldash_testlatency.mpd"

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Ending transfer for URL: "http://127.0.0.1:9000/lldash_testlatency.mpd"

lldash-relay: t=2025-11-17T11:27:25.346 event=request_received method=PUT url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:27:25.347 event=resource_chunk_received url=/v_0_0x0/v_0_0x0-2.m4s chunk_size=24
lldash-relay: t=2025-11-17T11:27:25.347 event=request_received method=PUT url=/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:27:25.347 event=resource_chunk_received url=/lldash_testlatency.mpd chunk_size=989
lldash-relay: t=2025-11-17T11:27:25.347 event=resource_created url=/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:27:25.347 event=request_completed method=PUT url=/lldash_testlatency.mpd status=200
lldash-relay: t=2025-11-17T11:27:25.347 event=connection_closed reason=client_closed
lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] notify end-of-stream.

lldash_packager: asynchronous debug message: [lldpkg::lldash_testlatency] [HttpSink (#1)] notify end-of-stream.

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Pipeline: flushing...

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Pipeline: flushed

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] Pipeline: completed

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] Pipeline: destroy

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [GPACMuxMP4 (#3)] Pipeline: destroy

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [2] Pipeline: destroy

lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [HttpSink (#1)] Pipeline: destroy

lldash-relay: t=2025-11-17T11:27:25.352 event=resource_created url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:27:25.352 event=request_completed method=PUT url=/v_0_0x0/v_0_0x0-2.m4s status=200
lldash-relay: t=2025-11-17T11:27:25.352 event=connection_closed reason=client_closed
lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Pipeline: destroy

testlatency: Sender finished with exit status: 0
testlatency: sender thread finished
lldash-relay: t=2025-11-17T11:27:25.443 event=resource_appeared url=/v_0_0x0/v_0_0x0-2.m4s waited_ms=100
lldash-relay: t=2025-11-17T11:27:25.443 event=resource_served url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:27:25.443 event=chunk_sent url=/v_0_0x0/v_0_0x0-2.m4s chunk_size=24
lldash-relay: t=2025-11-17T11:27:25.443 event=request_completed method=GET url=/v_0_0x0/v_0_0x0-2.m4s status=200
lldash-relay: t=2025-11-17T11:27:25.443 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:25.443 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:27:27.525 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:27:27.525 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:27.526 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:27:29.601 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:27:29.601 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:29.602 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:27:31.674 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:27:31.674 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:31.674 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:27:33.752 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:27:33.753 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:33.753 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash_play: nothing received for 10 seconds, assuming end of file
lldash_play: thread exiting
netdecoder: source.get returned no data
netdecoder: thread exiting
lldash-relay: t=2025-11-17T11:27:35.829 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:27:35.829 event=connection_closed reason=client_closed
lldash_play: asynchronous debug message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] wget: 'http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s'

lldash-relay: t=2025-11-17T11:27:35.830 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
netdecoder: stop
testlatency: receiver thread finished
testlatency: Stopping server thread...
testlatency: server thread finished
testlatency: count_total=98, count_lost_initial=0, count_lost_running=1, latency_ignored_count=4, latency_min=0.120, latency_max=0.332, latency_avg=0.216, latency_stddev=0.037
testlatency: Latency test passed.
lldash_play: calling lldplay_destroy()
lldash_play: asynchronous error: [lldplay::cwipc_lldplay] [lldplay_disable_stream] exception caught:failure HTTP download failed (url="http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s"): Server returned nothing (no headers, no data)


lldash_play: asynchronous info message: [lldplay::cwipc_lldplay] Pipeline: destroy

lldash_play: asynchronous info message: [lldplay::cwipc_lldplay] [stream #0] Pipeline: destroy

lldash_play: asynchronous info message: [lldplay::cwipc_lldplay] [DashDemuxer (#0)] Pipeline: destroy

lldash_play: lldplay_destroy() returned
((.venv) ) boor:lldash jack$ 

First thing I notice is the line

ldash-relay: t=2025-11-17T11:27:25.337 lldash_packager: asynchronous info message: [lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Manifest was not rewritten for on-demand and all files are being deleted.

Double-checked, and I am calling lldpkg_destroy() with flush=true.

Just after t=2025-11-17T11:27:25.352 my sender thread exits, so we can be sure that lldpkg_destroy() has returned. The last resource that was pushed just before this was the small one to url=/v_0_0x0/v_0_0x0-2.m4s.

It is subsequently passed to the receiver at t=2025-11-17T11:27:25.443.

Now we get to the "end game" with the receiver asking for url=/v_0_0x0/v_0_0x0-3.m4s which doesn't exist every 2 seconds, after which it gives up.

The Python code now closes the lldplay side, and during lldplay_destroy() we get a spurious error message

lldash_play: asynchronous error: [lldplay::cwipc_lldplay] [lldplay_disable_stream] exception caught:failure HTTP download failed (url="http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s"): Server returned nothing (no headers, no data)

[jackjansen]

Just realised that the logging output is much more interesting. Ran

LLDASH_LOGGING=1 python tests/testlatency/testlatency.py --npoints 100000 --fps 10 --seg_dur 8000 --duration 10 --octree_bits 6

Here is the output starting with the sender trying to close the connection:

sink_lldpkg: ts=2025-11-17T11:56:06.339 event=lldpkg_destroy_call url=http://127.0.0.1:9000/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:56:06.346 event=resource_chunk_received url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=14066
lldash-relay: t=2025-11-17T11:56:06.346 event=resource_created url=/v_0_0x0/v_0_0x0-1.m4s
lldash-relay: t=2025-11-17T11:56:06.346 event=request_completed method=PUT url=/v_0_0x0/v_0_0x0-1.m4s status=200
lldash-relay: t=2025-11-17T11:56:06.346 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:06.346 event=chunk_sent url=/v_0_0x0/v_0_0x0-1.m4s chunk_size=14066
lldash-relay: t=2025-11-17T11:56:06.346 event=request_completed method=GET url=/v_0_0x0/v_0_0x0-1.m4s status=200
lldash-relay: t=2025-11-17T11:56:06.346 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:06.347 event=request_received method=GET url=/v_0_0x0/v_0_0x0-2.m4s version=HTTP/1.1
lldash_play: ts=2025-11-17T11:56:06.350 event=lldplay_grab_frame_returned streamIndex=0 length=13926 url=http://127.0.0.1:9000/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:56:06.355 event=request_received method=PUT url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:56:06.356 event=request_received method=PUT url=/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:56:06.356 event=resource_chunk_received url=/v_0_0x0/v_0_0x0-2.m4s chunk_size=24
lldash-relay: t=2025-11-17T11:56:06.356 event=resource_chunk_received url=/lldash_testlatency.mpd chunk_size=989
lldash-relay: t=2025-11-17T11:56:06.356 event=resource_created url=/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:56:06.356 event=request_completed method=PUT url=/lldash_testlatency.mpd status=200
lldash-relay: t=2025-11-17T11:56:06.356 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:06.361 event=resource_created url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:56:06.361 event=request_completed method=PUT url=/v_0_0x0/v_0_0x0-2.m4s status=200
lldash-relay: t=2025-11-17T11:56:06.361 event=connection_closed reason=client_closed
sink_lldpkg: ts=2025-11-17T11:56:06.373 event=lldpkg_destroy_returned url=http://127.0.0.1:9000/lldash_testlatency.mpd
lldash-relay: t=2025-11-17T11:56:06.452 event=resource_appeared url=/v_0_0x0/v_0_0x0-2.m4s waited_ms=100
lldash-relay: t=2025-11-17T11:56:06.452 event=resource_served url=/v_0_0x0/v_0_0x0-2.m4s
lldash-relay: t=2025-11-17T11:56:06.452 event=chunk_sent url=/v_0_0x0/v_0_0x0-2.m4s chunk_size=24
lldash-relay: t=2025-11-17T11:56:06.452 event=request_completed method=GET url=/v_0_0x0/v_0_0x0-2.m4s status=200
lldash-relay: t=2025-11-17T11:56:06.452 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:06.452 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:56:08.533 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:56:08.533 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:08.534 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:56:10.617 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:56:10.617 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:10.618 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:56:12.704 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:56:12.704 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:12.705 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash-relay: t=2025-11-17T11:56:14.795 event=error_reply method=GET url=/v_0_0x0/v_0_0x0-3.m4s status=404 reason=not_found
lldash-relay: t=2025-11-17T11:56:14.795 event=connection_closed reason=client_closed
lldash-relay: t=2025-11-17T11:56:14.795 event=request_received method=GET url=/v_0_0x0/v_0_0x0-3.m4s version=HTTP/1.1
lldash_play: nothing received for 10 seconds, assuming end of file
netdecoder: source.get returned no data
testlatency: count_total=98, count_lost_initial=0, count_lost_running=1, latency_ignored_count=4, latency_min=0.216, latency_max=0.331, latency_avg=0.229, latency_stddev=0.012
testlatency: Latency test passed.
lldash_play: asynchronous error: [lldplay::cwipc_lldplay] [lldplay_disable_stream] exception caught:failure HTTP download failed (url="http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s"): Server returned nothing (no headers, no data)


lldash_play: ts=2025-11-17T11:56:16.418 event=async_log_message level=0 msg=[lldplay::cwipc_lldplay] [lldplay_disable_stream] exception caught:failure HTTP download failed (url="http://127.0.0.1:9000/v_0_0x0/v_0_0x0-3.m4s"): Server returned nothing (no headers, no data)

So, at ts=2025-11-17T11:56:06.339 we started shutting down the sender, and at ts=2025-11-17T11:56:06.373 this was completed.

Just before that, at ts=2025-11-17T11:56:06.350, the layout side got its last fragment.

Now nothing more happens on the receiving side until the timeout has happened.

[jackjansen]

Flash of insight!

Our API isn't good enough!

The only API call we use to check whether a new fragment is available is lldplay_grab_frame(self.handle, streamIndex, None, 0, None).

But a return value of 0 means there is no data available now. There is no way to distinguish that from and there won't be any more data available ever.

So, the questions are:

• Can lldash_playout/plugin.cpp determine the difference between these two conditions?
• How can we communicate this to the upper layers? My preference would be a new API call, maybe lldplay_eof() but I'm open to other ideas such as overloading the return value of lldplay_grab_frame.

[rbouqueau]

[lldpkg::lldash_testlatency] [MPEG_DASH (#0)] Manifest was not rewritten for on-demand and all files are being deleted.

Double-checked, and I am calling lldpkg_destroy() with flush=true.

This is different. Flush() means to process the sample in the pipeline. The log message is about keeping the files on the lldash-relay server.

lldash_play: nothing received for 10 seconds, assuming end of file

We discussed about updating the live MPD. At the moment we only download the MPD at the start of the session. This prevents us from exiting cleanly. We decided it was not an issue to timeout at exit.

Can lldash_playout/plugin.cpp determine the difference between these two conditions?

No, see my previous answer: it timeouts to exit.

How can we communicate this to the upper layers? My preference would be a new API call, maybe lldplay_eof() but I'm open to other ideas such as overloading the return value of lldplay_grab_frame.

We need to reload the MPD DASH manifest, detect the EOS, propagate it to the graph and change the API accordingly.

[jackjansen]

We need to reload the MPD DASH manifest, detect the EOS, propagate it to the graph and change the API accordingly.

Ok.

So we should decide when we want to reload the MPD. In my opinion it might be good to reload it whenever we got a 404 from trying to load a segment. This doesn't seem to be happening often anymore anyway (if at all).

Next question is who decides to reload the MPD. At user-level I don't have the information on a 404 having occurred, so should be somewhere in the lldash_playout plugin, or even lower?

And then the question of how to propagate this information to user level is (I think) just adding a flag in the lldash_playout handle and providing an API to get that flag.

Hmm, for that API: I wonder if we have more information "down in the plugin handle" that would be interesting to have access to at user level. For example how many items there are in the queue for each stream.

[rbouqueau]

So we should decide when we want to reload the MPD.

The MPD usually instructs on how often to reload :) But reloading on error is also a good idea.

for that API: I wonder if we have more information "down in the plugin handle" that would be interesting to have access to at user level. For example how many items there are in the queue for each stream.

Please make a list and we'll see what Signals allows to report. Signals heavily isolates modules - making it sometimes hard to get some data.

[jackjansen]

After discussion with @rbouqueau during the weekly teleconference he pointed out that we are killing the server before we are shutting down the receiver pipeline.

This is the reason we're getting this error. @jackjansen to fix.

[jackjansen]

It turns out exactly the same sequence of events happens in the VR2Gather/orchestrator production code. But here we are seeing it on the producer side.

When the session creator leaves the session the orchestrator terminates the session and kills the SFU.

All members of the session now get an indication that their session has disappeared, and they start closing all upload connections, and deleting everything, etc.

But the SFU is already dead, so the lldpkg pusher threads spend about one minute getting to grips with this fact:

TechnicalPlaygroundController: OnApplicationQuit: Leaving session.
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.
lldpkg: asynchronous warning: [lldpkg::pointcloud] [HttpSink (#1)] Transfer failed: Server returned nothing (no headers, no data)
.

I've asked for a stop-gap where the orchestrator keeps the SFU alive for a few seconds after the session has terminated, but it would be much cleaner if this wasn't needed, and if lldpkg would simply close the connection if this happens.

[jackjansen]

oops accidentally closed.