rtp_marker is missing when using proxy live555

[Apouchkal]

Hello,

we encounter an issue when we use live555 proxy for RTSP.
when use it the rtp_marker is always at 0.

we try without using live555 proxy on the same camera and it send us 0 and 1 when it need, but this info is always at 0 when we use the proxy.

thank you for your time.

Samuel

[RogerHardiman]

Hi
Thanks for letting me know that the live555 proxy does not set the marker bit.

The RTP standard says that we can use the marker bit OR we can look at the timestamp and if the timestamp of a RTP packet is different we know the 'last packet' was the one that should have had the marker bit set.

I've not implemented that in my SharpRTSP examples. It would need you to check the timestamp and then process the buffer of older RTP packets and then store this newly arrived RTP packet in the buffer.

I've no time at the moment to implement this but if you want to try and submit a Pull Request that would be great.

[Rakiah]

It seems like it does work but when the stream coming up from the proxy is consumed by RTSP Sharp client, it is very laggy, produce lot of artefacts and end-up crashing at some point, however when i use it with VLC there is absolutely no lags please see attached video
stream consumed by VLC
stream consumed by RTSP Sharp

Here was the code added into RTSP Sharp client that handle timestamps, it is very basic code that use the timestamp to compute the rtp_marker

                //the following code come directly from RTSP Sharp client, untouched
                int rtp_version = (e.Message.Data[0] >> 6);
                int rtp_padding = (e.Message.Data[0] >> 5) & 0x01;
                int rtp_extension = (e.Message.Data[0] >> 4) & 0x01;
                int rtp_csrc_count = (e.Message.Data[0] >> 0) & 0x0F;
                int rtp_marker = (e.Message.Data[1] >> 7) & 0x01;
                int rtp_payload_type = (e.Message.Data[1] >> 0) & 0x7F;
                uint rtp_sequence_number = ((uint)e.Message.Data[2] << 8) + (uint)(e.Message.Data[3]);
                uint rtp_timestamp = ((uint)e.Message.Data[4] << 24) + (uint)(e.Message.Data[5] << 16) + (uint)(e.Message.Data[6] << 8) + (uint)(e.Message.Data[7]);
                uint rtp_ssrc = ((uint)e.Message.Data[8] << 24) + (uint)(e.Message.Data[9] << 16) + (uint)(e.Message.Data[10] << 8) + (uint)(e.Message.Data[11]);
                
                //this is what I added to use the rtp_timestamp instead of the rtp_marker if it is available
                if (rtp_timestamp != 0) //basically it checks wether the timestamp is available, lots of cameras & system doesn't even send timestamp, which make it unreliable sometimes
                {
                    if (previous_timestamp == rtp_timestamp) //let's check against previous timestamp if it is equal
                        rtp_marker = 0; //if it is, setup rtp_marker as 0 so that it is considered incomplete paket
                    else
                    {
                        rtp_marker = 1; //otherwise, mark the packet as complete because we've been issued a new timestamp
                    }

                    previous_timestamp = rtp_timestamp; //let's assign previous one for next check
                }

Do you see any caveats in my code ?

I shall also mention that it is not the H264 decoding codec that is failing because taking the RTSP stream directly from RTSP Sharp server, without using live555 proxy will not produce any jittering effect & crashs

@RogerHardiman

[Rakiah]

nevermind, the code above is silly, because it does trigger that the packet is full, but it also add the first packet of the next frame into the previous frame, i'll try to modify that and see If i have better results

[Rakiah]

I've reworked the code and modified directly the Process_H264_RTP_Packet method in H264Payload.cs like this:

        uint previous_timestamp = 0;
        public List<byte[]> Process_H264_RTP_Packet(byte[] rtp_payload, uint rtp_timestamp) {
            
            //if its the very first packet, just buffer it and return
            if (previous_timestamp == 0)
            {
                temporary_rtp_payloads.Add(rtp_payload);
                previous_timestamp = rtp_timestamp;
                return null;
            }

            List<byte[]> nal_units = null;
            // now this check is actually for the previous packet, if the previous packet is complete because we've been issued a new timestamp
            // then process it and clear the buffer
            if (previous_timestamp != rtp_timestamp)
            {
                nal_units = Process_H264_RTP_Frame(temporary_rtp_payloads);
                temporary_rtp_payloads.Clear();
            }

            //if its not, just buffer it again
            temporary_rtp_payloads.Add(rtp_payload);
            previous_timestamp = rtp_timestamp;
            return nal_units;
        }

I've removed entirely the logic with the RTP Marker and only rely on timestamp now, it works nice when using Camera -> RTSPClient
using RTSPSharp -> RTSPClient
however using RTSPSharp -> Live555Proxy -> RTSPClient the problem remains, jittery camera and end up with a complete shutdown after a few seconds, I don't really know what I could do more with my code ATM, I don't see anything wrong with this piece of code..

[Rakiah]

More information about that, using this code it seems like some h264 frames are processed by
Process_H264_RTP_Frame(List<byte[]> rtp_payloads) in h264Payload.cs
but most of the frames that get processed actually have a nal_header_type of 0 (which is unknown)

I add also that VLC work smoothly with the tested RTSP stream and that the tested RTSP stream come from RTSPServer from SharpRtsp aswell

[rafcsoares]

Any improve in this issue @Rakiah

i'm getting similar problem from streams, probably related to nal_header_type 24 or 28 ...

Getting a bunch of artifacts in decoded stream

[RogerHardiman]

I'm back doing some SharpRTSP development so looking at this Issue Report properly now.
I tried a copy of Live555 proxy from 2018 and it used the Marker Bit correctly so I could not reproduce the issue.

I have just extracted the latest june 2020 copy of Live555, but it fails to build on my Mac. I'll move over to a Linux box.

Anyway, the fix to use the Timestamps is the right way to do things.
I'll test you code out

[rafcsoares]

@RogerHardiman my problem was fixed decoding the stream in another thread. When i was decoding stream in same thread of socket i miss some packages causing the artifact problems

Nothing related to H264Payload inself. Sorry for the false positive bug report.

Probably same problem with @Apouchkal?