Very long lag about camera frame display

[Aldarme]

Hi,

I'm using tello TT drone and i'm using "djitellopy2" to perform flight control on it.

However, I'm facing a very low fps rate when trying to just display real time stream frame from the camera drone.

To perform the display I'm using openCV and the exemple code to display stream frames.
Using:
"
from djitellopy import Tello
import cv2

myDrone = Tello()
myDrone.connect()
myDrone.streamon()
frameObj = self.myDrone.get_frame_read()

if frameObj.grabbed:
cv2.imshow("ARuco augmented", frameObj.frame)
cv2.waitKey(1)
"

Is there a known issue about this or not ?

Best regards,

[hildebrandt-carl]

I am also having this issue.

[zoldaten]

try to resize:

from djitellopy import tello
import cv2

me= tello.Tello()
me.connect()
print(me.get_battery())

me.streamon()

while True:
    img=me.get_frame_read().frame
    img=cv2.resize(img,(360,240))
    #img = cv2.flip(img,0)
    cv2.imshow("Image", img)
    cv2.waitKey(1)
    

[Aldarme]

I already try that solution, but that do not solve the problem.
I specifically have this issue with this API.
When I use the API from robomaster, I do not have such issue.
Best,

[hildebrandt-carl]

Yeah I have tried resizing too. The issue is that the self.frame inside the BackgroundFrameRead seems to always be 3-4 seconds behind. So if you just save the video to file, it looks smooth. However if you display it (any size) the frame you are displaying is 3-4 seconds behind.

[hildebrandt-carl]

@Aldarme where is the robotmaster API? It might help us isolate the issue if we can compare how they handle the video, compared to this library.

[ehong-tl]

Have you tried using the library straight from the latest master branch? Don't use the one from pypi or release as they are not the updated one.

[hildebrandt-carl]

Yes I am using the latest on master. I am not installing it from pypi.

I am using the latest with the updates that use BackgroundFrameRead and that uses pyav to decode the messages. It seems quite buggy and laggy. If you use it as it you have the 3-4 second lag. If you lower the framerate to 5FPS then pyav throws and exception that there are not enough frames, and you have to change some of the code to stop it from throwing that error (but that is a separate issue, and so I will not discuss it more here). I am mostly worried about the 3-4 second lag.

[Aldarme]

@hildebrandt-carl
Here it is:
https://robomaster-dev.readthedocs.io/en/latest/python_sdk/beginner_drone.html#initialize-the-robot

[hildebrandt-carl]

@Aldarme thanks, I will take a look at their implementation and see if I can figure out what they do to get rid of the lag.

[hildebrandt-carl]

Okay so I have a non lagging version. Most of the work is based off of the RoboMaster API. However I removed large amounts and made it work with my implementation. For anyone interested how is how you can get it to work:

You first need a VideoConnection class which handles the communication between the Tello Drone and your computer. It listens to the UDP sever and saves all the data inside a queue.

import time
import queue
import socket
import threading

class VideoConnection(object):

   def __init__(self):
       self._sock = None
       self._sock_queue = queue.Queue(32)
       self._sock_recv = None
       self._recv_count = 0
       self._receiving = False

   def __del__(self):
       if self._sock:
           self._sock.close()

   def connect(self, camera_ip: str="0.0.0.0", camera_port: int=11111):
       try:
           self._sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
           self._sock.bind((camera_ip, camera_port))
           print("OVER")
       except Exception as e:
           print("StreamConnection: connect addr {0}:{1}, exception {2}".format(camera_ip, camera_port, e))
           return False
       self._sock_recv = threading.Thread(target=self._recv_task)
       self._sock_recv.start()
       print("StreamConnection {0} successfully!".format(camera_ip))
       return True

   def disconnect(self):
       self._receiving = False
       self._sock_queue.put(None)
       if self._sock_recv:
           self._sock_recv.join()
       self._sock.close()
       self._sock_queue.queue.clear()
       self._recv_count = 0
       print("StreamConnection: disconnected")

   def _recv_task(self):
       self._receiving = True
       print("StreamConnection: _recv_task, Start to receiving Data...")
       while self._receiving:
           try:
               if self._sock is None:
                   break
               data, addr = self._sock.recvfrom(4096)
               if not self._receiving:
                   break
               self._recv_count += 1
               if self._sock_queue.full():
                   print("StreamConnection: _recv_task, sock_data_queue is full.")
                   self._sock_queue.get()
               else:
                   self._sock_queue.put(data)
           except socket.timeout:
               print("StreamConnection: _recv_task， recv data timeout!")
               continue
           except Exception as e:
               print("StreamConnection: recv, exceptions:{0}".format(e))
               self._receiving = False
               return

   def read_buf(self, timeout=2):
       try:
           buf = self._sock_queue.get(timeout=timeout)
           return buf
       except Exception as e:
           print("StreamConnection: read_buf, exception {0}".format(e))
           return None

Then you need a VideoHandler class. This class takes data from the queue generated by the VideoConnection and converts it using an h264 decoder.

import queue
import threading
import libmedia_codec

import numpy as np

from video_connection import VideoConnection

class VideoHandler(object):

    def __init__(self, camera_ip: str="0.0.0.0", camera_port: int=11111):

        # Init variables
        self._video_frame_count = 0
        self._video_streaming = False

        # Create the video queue
        self._video_frame_queue = queue.Queue(64)

        # Create the video decoder
        self._video_decoder = libmedia_codec.H264Decoder()

        # Turn on the video converter
        self._video_stream_conn = VideoConnection()
        self._video_stream_conn.connect(camera_ip, camera_port)
        self._video_decoder_thread = threading.Thread(target=self._video_decoder_task)
        self._video_decoder_thread.start()
        self._x = 0

    def _h264_decode(self, data):
        res_frame_list = []
        frames = self._video_decoder.decode(data)
        for frame_data in frames:
            (frame, width, height, ls) = frame_data
            if frame:
                frame = np.fromstring(frame, dtype=np.ubyte, count=len(frame), sep='')
                frame = (frame.reshape((height, width, 3)))
                res_frame_list.append(frame)
        return res_frame_list

    def _video_decoder_task(self):
        self._video_streaming = True
        print("_video_decoder_task, started!")
        while self._video_streaming:
            data = b''
            buf = self._video_stream_conn.read_buf()
            if not self._video_streaming:
                break
            if buf:
                data += buf
                frames = self._h264_decode(data)
                for frame in frames:
                    try:
                        self._video_frame_count += 1
                        if self._video_frame_count % 30 == 1:
                            print("video_decoder_task, get frame {0}.".format(self._video_frame_count))
                        self._video_frame_queue.put(frame, timeout=2)
                    except Exception as e:
                        print("_video_decoder_task, decoder queue is full, e {}.".format(e))
                        continue
        print("_video_decoder_task, quit.")

    def read_video_frame(self, timeout=3, strategy="newest"):
        if strategy == "pipeline":
            return self._video_frame_queue.get(timeout=timeout)
        elif strategy == "newest":
            while self._video_frame_queue.qsize() > 1:
                self._video_frame_queue.get(timeout=timeout)
            return self._video_frame_queue.get(timeout=timeout)
        else:
            print("read_video_frame, unsupported strategy:{0}".format(strategy))
            return None

You then can call the read_video_frame function with the strategy set to newest and you will always have the latest frame. I am now getting near perfect video performance, with no lag at all. One thing you will need to do is install libmedia_codec. To do that go on the RoboMaster API, go into the lib/libmedia_codec folder and install the python package using:

python3 -m pip install .

Best of luck.

[Aldarme]

@hildebrandt-carl
Hey, you produce an impressive work !
Can you consider doing a pull request about your work ?

[hildebrandt-carl]

Sure I have moved away from using this library and started writing my own. However it shouldn't take me too long to create a pull request for this library. I will try get it done before the end of the weekend.

Could you create that PR? @hildebrandt-carl

[steiraa]

Hello, sadly i cant build libmedia_codec,i get this error:

  Processing c:\ikarus\ansteuerung\000testgithubsol\robomaster-sdk\lib\libmedia_codec
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
  Building wheels for collected packages: libmedia-codec
  Building wheel for libmedia-codec (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for libmedia-codec (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [67 lines of output]
      running bdist_wheel
      running build
      running build_ext
      <string>:44: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      CMake Error at CMakeLists.txt:4 (project):
       Generator
       NMake Makefiles
        does not support platform specification, but platform
       x64
       was specified.
      -Configuring incomplete, errors occurred!
      See also "C:/Ikarus/Ansteuerung/000TestGithubSol/RoboMaster-SDK/lib/libmedia_codec/build/temp.win-amd64-cpython-310/Release/CMakeFiles/CMakeOutput.log".
      Traceback (most recent call last):
        File "C:\Users\ls15\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 351, in <module>
          main()
        File "C:\Users\ls15\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "C:\Users\ls15\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_vendor\pep517\in_process\_in_process.py", line 249, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\build_meta.py", line 413, in build_wheel
          return self._build_with_temp_dir(['bdist_wheel'], '.whl',
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\build_meta.py", line 398, in _build_with_temp_dir
          self.run_setup()
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\build_meta.py", line 484, in run_setup
          super(_BuildMetaLegacyBackend,
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 91, in <module>
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\wheel\bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\command\build.py", line 132, in run
          self.run_command(cmd_name)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\Users\ls15\AppData\Local\Temp\pip-build-env-3swhnhj9\overlay\Lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 49, in run
        File "<string>", line 77, in build_extension
        File "C:\Users\ls15\AppData\Local\Programs\Python\Python310\lib\subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', 'C:\\Ikarus\\Ansteuerung\\000TestGithubSol\\RoboMaster-SDK\\lib\\libmedia_codec', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=C:\\Ikarus\\Ansteuerung\\000TestGithubSol\\RoboMaster-SDK\\lib\\libmedia_codec\\build\\lib.win-amd64-cpython-310\\', '-DPYTHON_EXECUTABLE=C:\\Users\\ls15\\AppData\\Local\\Programs\\Python\\Python310\\python.exe', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY_RELEASE=C:\\Ikarus\\Ansteuerung\\000TestGithubSol\\RoboMaster-SDK\\lib\\libmedia_codec\\build\\lib.win-amd64-cpython-310\\', '-A', 'x64']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for libmedia-codec
Failed to build libmedia-codec
ERROR: Could not build wheels for libmedia-codec, which is required to install pyproject.toml-based projects

Any Help would be appreciated!

[M4GNV5]

@steiraa it first glance the error message does not support platform specification, but platform x64 looks like you are trying to specify to build for 64-bit windows, but the cmake file provided by libmedia_codec does not support that plattform

@hildebrandt-carl I would also greatly appreaciate a pull request with your video decoding code. I can merge it after testing, but I have to search for a working Tello battery first, they all seem to die after some weeks of not using/charging.