gvapython causing excessive delay in pipeline

[paakhim10]

Hello,
I have built a GStreamer pipeline for my use-case and recently added a callback for post-processing which is slowing down the inference.
My previous pipeline looked something like this:

gst-launch-1.0 filesrc location=${VIDEO_EXAMPLE} ! decodebin3 ! \
  videoconvert ! capsfilter caps="video/x-raw,format=BGRx" ! \
  gvadetect model=${MODEL} model_proc=${MODEL_PROC} device=CPU ! \
  gvafpscounter interval=1 ! \
  gvametaconvert ! \
  gvametapublish file-format=json-lines file-path=$METADATA_FILE ! \
  videoconvert ! x264enc ! mp4mux ! \
  filesink location=outputs/output.mp4

At this point, the fpscounter indicated an average fps of 58.7 fps. However, once I added my gvapython callback script, the average fps falls to 14.44 fps.

gst-launch-1.0 filesrc location=${VIDEO_EXAMPLE} ! decodebin3 ! \
  videoconvert ! capsfilter caps="video/x-raw,format=BGRx" ! \
  gvadetect model=${MODEL} model_proc=${MODEL_PROC} device=CPU ! \
  gvafpscounter interval=1 ! \
  gvapython module=assets/scripts/zone_detection.py ! \
  gvametaconvert ! \
  gvametapublish file-format=json-lines file-path=$METADATA_FILE ! \
  videoconvert ! x264enc ! mp4mux ! \
  filesink location=outputs/output.mp4

For additional information, I am also linking a metadata monitoring script to the pipeline through the shell script, which monitors the metadata.json file in real-time.

How can I bring the fps back up?
Thank you so much.

[brmarkus]

Have a look under "https://dlstreamer.github.io/dev_guide/python_bindings.html", specifically under the section "https://dlstreamer.github.io/dev_guide/python_bindings.html#performance-considerations".

Culd you try to build the pipeline in Python (instead of command-line and gst-launch-1.0 executable) and use callbacks (like in the example mentioned in the dev-guide linke above https://github.com/dlstreamer/dlstreamer/blob/master/samples/gstreamer/python/draw_face_attributes/draw_face_attributes.py )?
Or instead build the pipeline in a C/C++ application?

Have you tried to comment-out your processing code within the callback - as a sanity-check if the drop is caused by the the processing done in the callback, or caused by the "infrastructure" of GStreamerExecutable => DLStreamer => ContextSwitch => Python => PythonCallback =>DLStreamer => GStreamerExecutable ?

[paakhim10]

Hi, thanks so much for replying!
Replaced the gvapython code to

def process_frame(frame: VideoFrame):
    return frame

and the resultant fps is around 46 fps. It seems that a considerable amount of the delay is caused by the processing code, not the callback architecture. I am not sure if building the whole pipeline in Python will reduce the time if that's the case, could you tell me what you think of that?
Thanks!

[brmarkus]

From orignally ~58.7 fps to ~46 fps seems to be related to the "chain" from a console executable to DL-Streamer plugin to passing to a Python interpreter to running the callback and returning the same "chain" back to the gstreamer console application with several context switches.

You could build the same gstreamer pipeline (gstreamer and DL-Streamer) in Python - then your processing code is run in the same context.
Or you create a C/C++ application using gstreamer and DL-Streamer (or using OpenVINO APIs directly).

With respect to the processing code - you might be able to profile the code to find where the most time is spent, and then optimize it, or de-couple it or to e.g. redirect it to another process or write into a file and then processing the file offline?

[paakhim10]

Thank you for your advice! I did do that, simplified the processing a bit and managed to bring the ~14 fps speed to around 30 fps. For my use case, I am working on integration with WebRTC for frontend display of the stream, and the webrtcbin sink testing resulted in an even higher speed of ~46 fps.

On a side note, are there any resources in DLStreamer integration with WebRTC? I've been trying to use the backend as a signaling server and establish a connection between the frontend and the pipeline, and I've failed to do that. Truly appreciate your help!

[brmarkus]

You could even move your processing to a C/C++ library (and using e.g. OpenCL to delegate the code to GPU) and call that from within your implementation.

Are you sure with your input stream format the decoding and decoding pre- and post-processing is done using the GPU (if you have a GPU) to offload the CPU for your processing within your Python code?

E.g. the plugin videoconvert is always doing the conversion using CPU only! You could even adding GPU-zero-copy (surface-sharing) beween a GPU-video-codec and the (first) DLstreamer gva plugin (video/x-raw(memory:VASurface).
Also have a look under e.g. "https://dlstreamer.github.io/architecture_2.0/gstreamer_bins.html#video-pre-processing-backends-for-inference" and you could enable pre-processing to be done by DLStreamer (and also do it in the GPU).
Your processing in the plugin might be accelerated by using e.g. "numpy".

WebRTC seems very well supported by gstreamer itself...
You can find many posts and tutorials, like

• https://stackoverflow.com/questions/63946535/how-to-use-gstreamer-to-directly-stream-to-a-web-browser
• https://github.com/imdark/gstreamer-webrtc-demo