Metadata buffer not receiving user-defined Data Type (0x30) on RPi4 (raspiraw + FPGA bridge)

[mutantQ]

Hi, I know it’s not ideal to ask for help in a deprecated repo, but my team depends on raspiraw because it’s the only simple C-based rawcam app that’s easy to tinker with. Sorry in advance for bothering you here.

I’ve installed Buster on an RPi4 and managed to get raspiraw working — capture runs fine even with our FPGA bridge board (Trion T20). The problem is: we’re not receiving any metadata.
• We’re sending custom metadata using Data Type = 0x30 (user-defined byte-based, 0x30–0x37).
• We know 0x12 should be used for embedded data lines, but our FPGA cannot generate embedded lines and only passes them through directly.
• Based on prior issues and forum posts, any Data Identifier not matching the one in Unicam’s IDI0 (0x108) should be routed to the second buffer (flagged with MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO).
• However, packets appear to be dropped — console diffs between “with metadata” vs “without metadata” show no difference so far. I added a bunch of print statements, and you can check it out on the codingJang/raspiraw fork.
• I also noticed that bcm2835-unicam’s formats[] list doesn’t include 0x30–0x37, which makes me wonder if MMAL / Unicam simply doesn’t support user-defined byte-based Data Types.

Related question: In raspiraw’s callback(), can we rely on buffer ordering? We want to send alternating image data + metadata from our FPGA. From what I gather, vc.ril.rawcam produces two buffers per Frame End (same timestamp: one image, one metadata). Does that guarantee ordering across frames? In other words, will image/metadata callbacks always arrive in correct pairs, or can they interleave?

EDIT: I'm also interested in whether the metadata generated from
(a) the IMX219 sensor (with data type 0x12 and embedded data format code 0x0a), and
(b) the custom metadata we're sending (set to data type 0x30 and embedded data format code 0x0b)
can be mixed up / interleaved / etc.

Thanks a lot for your time, and for maintaining this tool that’s still really useful even now.

[6by9]

Hi, I know it’s not ideal to ask for help in a deprecated repo, but my team depends on raspiraw because it’s the only simple C-based rawcam app that’s easy to tinker with. Sorry in advance for bothering you here.

I’ve installed Buster on an RPi4 and managed to get raspiraw working — capture runs fine even with our FPGA bridge board (Trion T20). The problem is: we’re not receiving any metadata.
• We’re sending custom metadata using Data Type = 0x30 (user-defined byte-based, 0x30–0x37). • We know 0x12 should be used for embedded data lines, but our FPGA cannot generate embedded lines and only passes them through directly.
• Based on prior issues and forum posts, any Data Identifier not matching the one in Unicam’s IDI0 (0x108) should be routed to the second buffer (flagged with MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO).

Correct

• However, packets appear to be dropped — console diffs between “with metadata” vs “without metadata” show no difference so far. I added a bunch of print statements, and you can check it out on the codingJang/raspiraw fork.

No idea on that, and won't investigate on a deprecated interface and app.

• I also noticed that bcm2835-unicam’s formats[] list doesn’t include 0x30–0x37, which makes me wonder if MMAL / Unicam simply doesn’t support user-defined byte-based Data Types.

There is no defined V4L2 format that would use data types 0x30-0x37, therefore there will be no mapping entry in bcm2835-unicam's format list.

Related question: In raspiraw’s callback(), can we rely on buffer ordering? We want to send alternating image data + metadata from our FPGA. From what I gather, vc.ril.rawcam produces two buffers per Frame End (same timestamp: one image, one metadata). Does that guarantee ordering across frames? In other words, will image/metadata callbacks always arrive in correct pairs, or can they interleave?

Yes you should always get buffers alternating between MMAL_BUFFER_HEADER_FLAG_FRAME_END and MMAL_BUFFER_HEADER_FLAG_FRAME_END | MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO. The code literally makes the two buffer complete callbacks sequentially, triggered by the frame end interrupt. The core keeps completed buffers in order.

EDIT: I'm also interested in whether the metadata generated from (a) the IMX219 sensor (with data type 0x12 and embedded data format code 0x0a), and (b) the custom metadata we're sending (set to data type 0x30 and embedded data format code 0x0b) can be mixed up / interleaved / etc.

How are you merging the two streams?

[mutantQ]

No need to investigate, thanks for responding.

How are you merging the two streams?

I'm using the hardened MIPI CSI-2 block on the Trion T20 FPGA board, but I couldn't find documentation on its behavior.
I uploaded the question in Efinix (the manufacturer's) forum yesterday, so will get response in a day or two. I will get back to you by then.
Thanks for helping out!

[mutantQ]

Still no reply from Efinix - I'll get back to you as soon as they reply (or when I come up with a solution).