HEVC/h265 support for GStreamer on Raspbery Pi 4B/400/5

[fduncanh]

Now that UxPlay 1.70 supports HEVC/h265/4K video, this issue thread is for discussion of how to get the stateless Video4Linux2 HEVC driver(s) on Raspberry Pi 4B/400/5 working with the stateless V4l2 plugin in gstreamer1.0-plugins-bad

• this continues discussion in Trying to add h265 support (for modern clients with wired connection) (was *** ERROR: unsupported h265 video detected) #233

LibreELEC/Kodi have HEVC on these Pi's working using a modified ffmpeg created by R Pi engineers:
see jc-kynesim/rpi-ffmpeg#94 (comment)

and https://discourse.gstreamer.org/t/v4l2-stateless-api-support/3339

• The gstreamer code is at https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-plugins-bad/sys/v4l2codecs?ref_type=heads

• see here for useful info from LibreELEC
  https://wiki.libreelec.tv/hardware/raspberry-pi

• if we can understand how to access the HEVC drivers on these R Pi models working with UxPlay , perhaps with a patch to gstreamer-plugins-bad, a pull request to gstreamer can be made (this was done in the past to get GPU Hardware H264 decoding on R Pi 4B and earlier working with UxPlay (with the stateful v4l2 plugin in gstreamer-plugins-good).

• The stateless V$L2 API is described below: hopefully this is already mainly implemented by gstreamer
  https://www.linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/dev-stateless-decoder.html

[connorh315]

Just as a minor bit of information. I've nearly gotten ffmpeg working with uxplay - Currently the frames are all green for some reason but I'll iron that out soon. Main thing is though that the decoder block on the RPi 4 is capable of decoding the AirPlay H265 packets and it seems to run at a relatively good frame rate (hard to tell as the stream looks shocking). The relevant changes just needs to be pushed into gstreamer now.

[fduncanh]

I have some RPI 5's and can dedicate one to testing too.

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

Stateless v4l2 justs queues an encoded frame in the "OUTPUT" (from encoder) queue
and dequeues if from the "CAPTURE" queue, so interaction with the driver which connects to the hardware is
relatively simple. I just dont yet know how the interface to the driver works, and where it is in the ffmpeg code.

[fduncanh]

more info from jc-kinesim

The driver is in the PiOS linux tree - you can find it under driver/staging/media/rpivid/*. Hevc decode is a completely independent h/w block - GPU not involved. I'm sure the h/w is proprietary, I'm not completely sure to whom, so you are on your own for figuring out what is going on there (but honestly the h/w setup corresponds quite closely to the info passed in the V4L2 stateless structures so most of it is quite obvious).

[fduncanh]

current R Pi OS HEVC driver

https://github.com/raspberrypi/linux/tree/rpi-6.6.y/drivers/staging/media/rpivid

[connorh315]

Ideally we would rebuild the RPi ffmpeg from its code so printf statements could be inserted to follow how it connects to the drivers and to which drivers).

RPi-FFmpeg is quite easy to build, unfortunately it just takes quite a while (~30 minutes on a RPi 4 with a decent SD card). I expect that subsequent rebuilds with minor changes would be faster but I haven't tried. Just something worth mentioning is that the hardware decoder outputs a DRM-frame which doesn't seem to be supported by many graphics libraries. I originally used SDL and tried to copy the hardware frame via the CPU into main memory to then be converted to YUV and eventually output back onto the display however it seemed to make those green frames that I was talking about earlier. Instead I'm using a OpenGL instance now, as I'm relatively sure that it can directly render DRM-Prime frames without any conversion/copying. I just want to get a working prototype at the moment and test that it can decode relatively well (it seemed relatively stable with an iPhone 15 Pro, it was decent with an iPad pro and i'll test my macbook later once I've actually got it stably outputting frames (can force the full 3840x2160 output with that) to stress test it.

[connorh315]

Might have to give up with trying to get ffmpeg imported. Non-stop getting green-tinted frames from the decode and I can't seem to figure out what I'm doing wrong. Including ffmpeg into uxplay is relatively easy though (including adding hw-accelerated). Very frustrating

[fduncanh]

I am trying to find out why gstreamer is not detecting the rpivid driver on a R Pi 5.

The rpivid_hevc driver is loaded (see it in lsmod), the gstreamer libv4l2codec is correctly compiled and present (installed by gstreamer-1.0-plugins-bad)

"gstreamer-inspect-1.0 v4l2codec" shows the drivers, but lists "0 features" which means that no v4l2 devices was detected.

however, the v4l2 devices can be listed with "v4l2-ctl --list-devices" (v4l2-ctl is in the "v4l-utils" package, which is probably installed, if not, install with apt on R Pi OS)

This shows that the rpivid kernel model (on R Pi OS) is connected to /dev/video19 on my system. (and /dev/media0)

"v4l2-ctl -d19 -D" lists all its details, including that it is enabled.

[fduncanh]

posted a query about this on GStreamer discourse board: deleted it

$v4l2-ctl --list-devices
<snip>
rpivid (platform:rpivid):
        /dev/video19
        /dev/media0

$v4l2-ctl -d19 -D
Driver Info:
        Driver name      : rpivid
        Card type        : rpivid
        Bus info         : platform:rpivid
        Driver version   : 6.6.51
        Capabilities     : 0x84204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04204000
                Video Memory-to-Memory Multiplanar
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : rpivid
        Model            : rpivid
        Serial           : 
        Bus info         : platform:rpivid
        Media version    : 6.6.51
        Hardware revision: 0x00000000 (0)
        Driver version   : 6.6.51
Interface Info:
        ID               : 0x0300000c
        Type             : V4L Video
Entity Info:
        ID               : 0x00000001 (1)
        Name             : rpivid-source
        Function         : V4L2 I/O
        Pad 0x01000002   : 0: Source
          Link 0x02000008: to remote pad 0x1000004 of entity 'rpivid-proc' (Video Decoder): Data, Enabled, Immutable

[fduncanh]

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

[connorh315]

This is where the problem with gstreamer happens

0:00:00.143873339 82483 0x55563a41b400 INFO              v4l2codecs gstv4l2codecdevice.c:391:gst_v4l2_codec_find_devices: Found decoder device rpivid-proc
0:00:00.144000691 82483 0x55563a41b400 INFO              v4l2codecs plugin.c:59:register_video_decoder:<v4l2decoder0> Registering rpivid-proc as H265 Decoder
0:00:00.144019024 82483 0x55563a41b400 WARN      v4l2codecs-h265dec gstv4l2codech265dec.c:1684:gst_v4l2_codec_h265_dec_register: Not registering H265 decoder since it produces no supported format
0:00:00.144033524 82483 0x55563a41b400 INFO      GST_PLUGIN_LOADING gstplugin.c:987:_priv_gst_plugin_load_file_for_registry: plugin "/lib/aarch64-linux-gnu/gstreamer-1.0/libgstv4l2codecs.so" loaded

its a place to start to see what is going wrong.

I was reading somewhere that the output of the decoder is SAND8/SAND30... Maybe that's where the issue lies?

[fduncanh]

The problem is much more basic than that:

   src_caps = gst_v4l2_decoder_enum_src_formats (decoder);

  if (gst_caps_is_empty (src_caps)) {
    GST_WARNING ("Not registering H265 decoder since it produces no "
        "supported format");
    goto done;

• the ioctl calls to VIDIOC_G_FMT and VIDIOC_ENUM_FMT (to interrogate the decoder) dont fail, but the caps that gst_v4l2_decoder_enum_src_formats(decoder) extracts from the results and then returns are EMPTY when converted to a string with gst_caps_to_string(caps) This causes rpivid to be rejected as a v4l2 decoder.

• I need to find out what is in the responses to the two v4l2 calls. (which means learning about them)

• It looks like they are being analyzed for things the driver cant yet know about (pixelformat, width, height), maybe its an error for this code to be used as a registration test when gst-inspect-1.0 is run, ???? (i'll try commenting out the test)

      gst_v4l2_decoder_probe_caps_for_format (self,
      fmt.fmt.pix_mp.pixelformat, fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);

GstCaps *
gst_v4l2_decoder_enum_src_formats (GstV4l2Decoder * self)
{
  gint ret;
  struct v4l2_format fmt = {
    .type = self->src_buf_type,
  };
  GstCaps *caps;
  gint i;

  g_return_val_if_fail (self->opened, FALSE);

  ret = ioctl (self->video_fd, VIDIOC_G_FMT, &fmt);
  if (ret < 0) {
    GST_ERROR_OBJECT (self, "VIDIOC_G_FMT failed: %s", g_strerror (errno));
    return FALSE;
  }

  caps =
      gst_v4l2_decoder_probe_caps_for_format (self,
      fmt.fmt.pix_mp.pixelformat, fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);

  /* And then enumerate other possible formats and place that as a second
   * structure in the caps */
  for (i = 0; ret >= 0; i++) {
    struct v4l2_fmtdesc fmtdesc = { i, self->src_buf_type, };
    GstCaps *tmp;

    ret = ioctl (self->video_fd, VIDIOC_ENUM_FMT, &fmtdesc);
    if (ret < 0) {
      if (errno != EINVAL)
        GST_ERROR_OBJECT (self, "VIDIOC_ENUM_FMT failed: %s",
            g_strerror (errno));
      continue;
    }

    tmp = gst_v4l2_decoder_probe_caps_for_format (self, fmtdesc.pixelformat,
        fmt.fmt.pix_mp.width, fmt.fmt.pix_mp.height);
    caps = gst_caps_merge (caps, tmp);
  }

  return caps;
}

[fduncanh]

HOORAY!!!!

commenting out that test indeed allows the "v4l2slh265dec V4L2 Stateless H.265 Video Decoder" to be registered, and show up with "1 feature" in "gst-inspect-1.0 v4l2codec"

[connorh315]

And it works???

[fduncanh]

not yet, sorry
GST_DEBUG=2
The caps are returned as "empty"

v4l2slh265dec0 error Unsupported bitdepth/chroma format
v4l2slh265dec0 error No support for 2896x2160 8bit chroma IDC1
v4l2slh265dec0 Failed to negotiate with downstream
v4l2slh265dec0 subclass does not want accept new sequence
v4l2slh265dec0 Failed to process sps
queue error internal data stream error
queue error streaming stopped, reason not-negotiated (-4)

but it's a start......