Permanent hang in video stream if FPS drops to 0 for too long

[connorh315]

Receiver device running UxPlay: Raspberry Pi 4
Sender: iPad Pro (M1)
OS Version: Most recent Raspberry Pi OS (64-bit) with Desktop - "Debian GNU/Linux 12 (bookworm)" - released 2024-07-04
GStreamer version - 1.22.0
UxPlay version: 1.69
Launch command: ./uxplay -p -d
Videosink: xvimagesink

Issue:

Apps like Goodnotes can produce streams that at times will send 0 FPS as there are no changes on the screen. When this 0 fps state has happened for more than 5 seconds the on-screen video will become temporarily unresponsive, where new packets will be received but the screen will stay stuck and will not update with the new packets. Sometimes it can recover if it has not been left in this 0fps state for too long. However, if the sender sends 0 FPS for too long, it will become permanently stuck and nothing can be done to recover the stream. The console clearly shows that the sender will send new packets to update the screen with but the screen will not change. This issue does not occur with -avdec, and only seems to occur with xvimagesink as the issue disappears when using ximagesink (granted that the stream is much more laggy as ximagesink doesn't seem to be as good).

[connorh315]

Interestingly, -vsync no fixes this issue - it also seems to make the stream a lot smoother (not just in this app, but in general). Please let me know if i'm missing something and I'll close this, however I doubt that the stream hanging is intended behaviour.

[thiccaxe]

Possibly related to #207

[connorh315]

It looks like it

[fduncanh]

if its related to #207, the earlier fix for #207 got broken in gstreamer-1.24.x, created an urgent need for a revised fix.

since none was found , the earlier fix is just omitted if gstreamer .+ 1.24 is used.

So this needs a solution of some kind.

EDIT: you are using gstreamer-1.22.0

-vsync keeps the audio and video in sync using their timestamps.

This will drop video frames if they can't be matched to audio.
to see if this is happening use "export GST_DEBUG=2" before running uxplay.~~

"-vsync no" uses a different clock method best for "live streaming" , and ignores timestamps.

[fduncanh]

so to see this, one just runs Goodnotes 6 on an ipad for X? minutes with no screen input? with mirror to r pi 4?

[connorh315]

That should do it, I'll check with the export GST_DEBUG=2 now

[connorh315]

Once I've waited for a few seconds, and then attempt to make the screen change (as such triggering packets to be sent), the console gets spammed with this:
0:00:33.019607473 4611 0x7f800808c0 WARN videodecoder gstvideodecoder.c:3668:gst_video_decoder_clip_and_push_buf:<v4l2h264dec0> Dropping frame due to QoS. start:0:00:32.697566674 deadline:0:00:32.697566674 earliest_time:0:00:35.634703500 for each packet that gets received

[fduncanh]

you definitely should be using "-vsync no" for any live streaming such as goodnotes.

The README section on running uxplay explains it:

The older method which does not drop late video frames worked well on more powerful systems, and is still available with the UxPlay option "-vsync no"; this method is adapted to "live streaming", and may be better when using UxPlay as a second monitor for a Mac computer, for example, while the new default timestamp-based method is best for watching a video, to keep lip movements and voices synchronized. (Without use of timestamps, video will eventually lag behind audio if it cannot be decoded fast enough: hardware-accelerated video-decoding helped to prevent this previously when timestamps were not being used.)

The default is -vsync since most people are probably not "live streaming", but you are, with goodnotes etc.

[fduncanh]

Should the readme be more explicit?

[thiccaxe]

I looked at the gstreamer source code, and in general, video sinks don't seem to have any specific handling code when gstreamer sends a PLAYING_TO_PAUSED or PAUSED_TO_PLAYING event. So no clue why a specific sink works or not.

[connorh315]

I'm not too sure that this is intended though, as on Windows the problem does not exist using the directx videosinks, also the video sink ximagesink does not have the same problem either. It's just the xvimagesink that seems to cause this issue. Say it can be circumvented like you say by using -vsync no but I do wonder why like thiccaxe said it works for some sinks but not for others.

[thiccaxe]

We can try to write some sample code to see if xvimagesink can be paused or not, using something like gst-launch-1.0 -v videotestsrc ! xvimagesink etc.

maybe xvimagesink is causing the issue. Nevertheless, the current code for handling the 0-fps state is not the best.

[thiccaxe]

can we try adding the timeoverlay to gain some more insight about exactly what is happening when the stream is paused or resumed?

[thiccaxe]

So basically I think this is what happens

Gstreamer Internal Clock | Packet timestamp

[start]

1 | 1
2 | 2
3 | 3

[turn off phone screen for 5 seconds then turn back on]

8 | 4 -> videodecoder drops (https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-base/gst-libs/gst/video/gstvideodecoder.c#L3665) -> gstreamer does not increment time (https://gitlab.freedesktop.org/gstreamer/gstreamer/-/blob/main/subprojects/gst-plugins-base/gst-libs/gst/video/gstvideodecoder.c#L3190)
8 | 5 -> " "
8 | 6 -> " "
8 | 7 -> " "
8 | 8 -> forwards packet

so either we need to stop the gstreamer clock, or apply a delta to the packet timestamps