MQTT maximum throughput

[Elfelsoufim]

Hello,

For my application, I need to send messages from an ESP32 to my main computer (which hosts a mosquitto broker) at a minimum rate of 1000 messages per second. However, when I tested Smooth MQTT throughput using the example found in Smooth/test/mqtt, I get the following result:

Transmission Rate:

5 messages per second for QoS = Exactly once.
10 to 12 messages per second for QoS = At least once. Same for QoS = At most once.

That, however, is not the expected behaviour, since:

1. ESP32 allegedly can send about 2000 messages per second. https://www.reddit.com/r/esp8266/comments/5trsvl/esp32_wifi_real_life_throughput/ddokmz6/?utm_source=share&utm_medium=web2x&context=3.
2. Mosquitto broker was tested to be able to handle 7000 messages per second. http://rexpie.github.io/2015/08/23/stress-testing-mosquitto.html.

For troubleshooting, I have tried several solutions. Notably:

• I tried deactivating the power save mode. This had no noticeable effect on the throughput.
• I tried increasing the buffer size. This only increased the latency.

To detect the source of the problem, I tried timing several methods such as publish(), publish_next() and send_packet(). All of these take at most a few milliseconds to execute.
I also tried timing the loop inside exec(). Here I found that it sometimes takes up to 150 ms to execute the loop. Knowing that, I tried circumventing exec() by not calling start() (start() calls exec()). I did this by calling init() and then send_message() repeatedly. Please see the code provided below.

After trying this fix, the throughput did not improve at all, which leads me to think that send_message() (which really is just publish()) is guilty of this issue. But considering that publish() takes only a few milliseconds to put the data in the transmit buffer, and after having read this #142, I conclude that there must be a problem with how messages are being sent.

What I really want to ask here is:

1. Am I using "MqttClient.cpp" wrong?
2. What is the expected throughput of MQTT client? Is there a benchmark?
3. How could I increase the maximum transmission rate of "MqttClient.cpp"?

For reference, here is the environment I used:

• Development Kit: ESP32-DevKitC
• Kit version: v4
• Module or chip used: ESP32-WROVER-E 8MB
• IDF version (run git describe --tags to find it): v4.2.1
• Build System: CMake|idf.py
• Compiler version (run xtensa-esp32-elf-gcc --version to find it): (crosstool-NG esp-2020r3) 8.4.0
• Operating System: Windows
• (Windows only) environment type: ESP Command Prompt
• Using an IDE?: Yes, VSCode
• Power Supply: USB

Here is the code I used to test MQTT: GIST

[PerMalmberg]

Hello,

1. I don't think you are.
2. No, it was fast enough for my purposes, but I remember being able to send quite a lot of messages per second. Have you tried the test/example?
3. I wish I could tell you, but I don't have an answer. It could be as simple as moving the ESP32 to another location, I remember doing that and getting much better Wifi speeds.

[squonk11]

I had quite interesting speed improvement when I reduce the timeout for select from 10ms to 1ms here.
Also my feeling is that the there is somewhere a delay when processing messages between tasks. If I interpret this code right then the tick() function is not called when a message arrived; tick() will then be called during next Task cycle. Here it is important which is the Task cycle time for your MQTT task. Maybe it could make sense to put a call to tick() also in the else path of if (!queue_ptr.owner_before(empty_ptr) && !empty_ptr.owner_before(queue_ptr))in Task::exec()?

[PerMalmberg]

It does wait for data to become available, @squonk11

Smooth/lib/smooth/core/Task.cpp

Line 164 in c874fed

auto queue_ptr = notification.wait_for_notification(tick_interval);

[squonk11]

yes, I agree, but in case there is a message, then only the message is being processed and tick() is not being called (if I interpret the code correctly).

[PerMalmberg]

Correct - but tick() isn't meant to handle events (i.e. incoming messages). Tick is meant to do things that are not triggered by an event.

The Task impl. attempts to ensure that the tick is called on the desired interval.

Smooth/lib/smooth/core/Task.cpp

Line 148 in c874fed

if (tick_interval.count() > 0 && delayed.get_running_time() > tick_interval)

[squonk11]

ah, o.k. I understand.

[squonk11]

@Elfelsoufim : The link in your post seems to point to a wrong location?
But two comments from my side:

1. The test done there is made using a simple ping/pong protocol in polling mode via UDP. That probably isn't comparable with a protocol like MQTT via TCP runing on a cyclic task.
2. The network infrastructure you are using can also make a difference. A direct connection between PC and ESP32 will give better results as if the telegrams pass e.g. a WiFi router, a WiFi Repeater, an ethernet switch etc.

[Elfelsoufim]

Hello, thank you both for your timely support.

@squonk11 I have tried all the solutions you have suggested both in this issue and in #142, without success, unfortunately. My transmission rate seems to be fixed at around 12 messages per second.

I ran a few tests to benchmark Smooth MQTT. I put my results and my conclusion in this short document:
Benchmark MQTT.docx
Benchmark MQTT.pdf

The tests and documentation are rather informal, but I believe they illustrate my main point: Smooth's MQTT can be optimized further 😄.

[PerMalmberg]

Serious investigation :)

Yes, I'm sure there are ways to optimize it. I have a faint memory of the parsing of incoming messages being a bottleneck, iirc it was read a byte at a time to determine the message size etc. I don't remember if I optimized that part or not. so that may be a starting point.

[Elfelsoufim]

Hello,

I have tested ESP-IDF's MQTT library using very similar code & conditions to the TCP/IP benchmark I have done previously to give us a useful comparison with Smooth MQTT. Here is the documentation:
Benchmark ESP_IDF_MQTT.pdf

The result is an average of 640 messages sent per second. I am only a student so it is not something I could do myself, but this would be a nice result to achieve with Smooth MQTT.