Velodyne: Pointclouds do not get published if hardware goes through a power cycle when udp_only is false

[mbharatheesha]

I see this on a VLP16 setup. Starting the driver with launch_hw=true and udp_only=false brings up the driver as expected and pointclouds are regularly published on /velodyne_points topic, along with relevant diagnostics also available.

However, if the hardware goes through a power cycle and comes back alive again (confirmed by pinging the LiDAR IP from host), pointclouds aren't published anymore on /velodyne_packets. The decoder component keeps publishing the "Missed pointcloud deadline warning" every 5s. The only way to recover from this seems to be a restart of the entire node.

When udp_only=true, the pointclouds start appearing a couple of seconds after the hardware comes back up online as expected and a node restart is not required. While this can be an interim solution, running with diagnostics might not be desirable.

[mojomex]

Thank you for the feedback!

Sounds like the TCP socket blocks forever on connection loss, whereas the desired behavior is for it to time out and re-connect as soon as possible. It's not immediately clear from the code why this would affect pointcloud output though.

As Nebula is developed primarily with autonomous driving as a use case, recovering from sensor power loss has not been a priority for us, so there is no logic covering re-connection once the sensor recovers. If this is something your use case requires, we could work together on coming up with a fix.

[mbharatheesha]

@mojomex Thanks for your response.

A further update. I managed to get my hands on a Hesai LiDAR and there I notice something more dramatic.

What is happening is, the pointclouds do start coming through, but then a runtime error gets thrown with
[hesai_ros_wrapper_node-1] [ERROR] [1745580714.797696049] [TcpSocket::asyncReceiveHandlerHeaderPayload]: Connection reset by peer [hesai_ros_wrapper_node-1] [ERROR] [1745580715.797847132] [hesai_ros_wrapper_node.HwInterface]: 0x09 (0) Request did not finish within 1s [hesai_ros_wrapper_node-1] terminate called after throwing an instance of 'std::runtime_error' [hesai_ros_wrapper_node-1] what(): Communication error: Request timeout
I tried to do some code backtracing here and in the powercycle scenario, the acquisition of the timed mutex here is failing. And perhaps because the send_receive() function returns an error due to this failed attempt at getting the mutex, a value_or_throw() gets called somewhere down the line triggering the exception and exit.

I am not completely conversant with this part of the code. But I am happy to help with any tests (and implementation) if you have some possible thoughts on what could be going wrong.

[mojomex]

@mbharatheesha Thanks for your investigation.

As Nebula is developed primarily with autonomous driving as a use case, recovering from sensor power loss has not been a priority for us, so there is no logic covering re-connection once the sensor recovers.

This also applies to Hesai sensors. We are aware that the ideal way to handle this would be to recover, but for autonomous driving we would have to introduce a state machine + diagnostics to inform the vehicle of the error. A big diagnostics overhaul (mainly only for Hesai at this point, I'm afraid) is scheduled for next month.

From a technical perspective, instead of throwing, an appropriate retry mechanism should be introduced. However, this mechanism might not be the same for all PTC commands (e.g. we need to retry until successful when setting sensor config, but it might be okay to not retry at all for diagnostics statuses).

As for the Velodyne issue, I'll have to do that once I have access to hardware again, which is in about 3 weeks. If you have any more insights until then, I'll be happy to help where possible!