fix: lower packet loss to near-zero levels

[mojomex]

PR Type

• Bug Fix

Related Links

• [TIER IV internal link] -- internal ticket

Description

This PR lowers the chance of packet loss occurring for all supported LiDARs by minimizing the number of mutex locks in the UDP receiver thread.

The UDP receiver thread was losing packets before, mainly because of scheduling:

1. Decoder is currently reading from queue
2. UDP receiver cannot acquire lock, its thread gets suspended
3. N UDP packets arrive (and are lost) while thread is not scheduled
4. When thread gets finally scheduled, it has lost packets

This is mostly mitigated by moving the mt_queue buffer from between the hardware interface and decoder to between the decoder and pointcloud/packets publishers. In other words, the queue is moved from the high-frequency part of Nebula (up to 36kHz for OT128) to the low-frequency part (10Hz or whatever the sensor framerate is set to).

The below figure shows the number of points in an unfiltered pointcloud (as a proxy for the number of packets received) for the baseline and fixed ("queue_only") implementations. The sensor tested was OT128 in dual return & high resolution mode.
The benchmarking setup was as follows (every run was repeated 3 times):

• taskset Nebula to cores 0-7
• taskset ros2 bag record -s mcap --max-cache-size 5000000000 /pandar_points to cores 8-11
• run 8 instances of ros2 topic echo /pandar_points --field header taskset to core 12 + 2i for instance i
• stop all processes after 60s

The plot was generated from the concatenated rosbags.

	messages	mean	std	min	0.1%	1%	10%	50%
baseline	1715	920577	2436.64	865024	895010	911908	919040	921600
fix	1717	921585	582.054	897536	920503	921600	921600	921600
baseline (% lost)		920577	0.11	6.14	2.89	1.05	0.28	0
fix (% lost)		0.00		2.61	0.12	0	0	0

ℹ️ This PR applies to all Hesai, Velodyne, and Robosense sensors.
ℹ️ #169 (Aeva Aeries II) already uses a similar approach and should be immune to packet loss
⚠️ I am not sure at the moment whether this change would benefit Continental sensors as well (HW monitor and other functions handled in decoder, packet rate unknown to me), so I skipped them for now. @knzo25 have you observed packet loss on Radars so far?

Pre-Review Checklist for the PR Author

PR Author should check the checkboxes below when creating the PR.

• Assign PR to reviewer

Checklist for the PR Reviewer

Reviewers should check the checkboxes below before approval.

• Commits are properly organized and messages are according to the guideline
• (Optional) Unit tests have been written for new behavior
• PR title describes the changes

Post-Review Checklist for the PR Author

PR Author should check the checkboxes below before merging.

• All open points are addressed and tracked via issues or tickets

CI Checks

• Build and test for PR: Required to pass before the merge.

[mojomex]

This was not needed after all and the same can be achieved by allocating larger network buffers (net.core.rmem_default).