Flexbe freezing network on remote PC

[RPRKLR]

Description:

When I start up flexbe_app and flexbe_behavior_engine on my local PC, everything works fine with no issues regarding the connection. However, when I start up the flexbe_onboard on our robot (which is on the same Ethernet network), and then start up the flexbe_app_ocs, the network freezes for 2 minutes. We lose connection to the device and all network connections, including internet browsing.

Steps to Reproduce:

1. Start flexbe_onboard on the robot, which is on the same Ethernet network.
2. Start flexbe_app.

Expected Behavior:
The network should remain stable, and all connections should work without interruption.

Actual Behavior:
The network freezes for 2-5 minutes, causing loss of connection to the device and all networks, including internet browsing.

We can not provide any screenshots or logs for this occurrence, since everything is frozen on the robot, but after some time the network connection stabilize itself, and we can communicate with the robot.

Additional note
We also experienced that when we start up the app, the network outgoing traffic jumps to 900MBit/s

Environment:

• OS: Ubuntu 22.04.3
• ROS2 Version: Iron
• Network: Ethernet

[dcconner]

This is a new one to me.

We just made the new flexbe_webui live; you might try it instead of the flexbe_app.

Questions:

Is this on initial launch before running behavior, or is it on start of a behavior?
Maybe start onboard and verify you can get data prior to starting the OCS side.
ros2 topic echo /flexbe/heartbeat to see if all is well prior to launching OCS.

What happens if you start OCS but not onboard?

There are not any fancy web comms between onboard and OCS. It is all basic ROS topics.

[dcconner]

We also experienced that when we start up the app, the network outgoing traffic jumps to 900MBit/s

Hmm. The only traffic should be basic ROS messaging between OCS and onboard.
Try starting OCS by itself and then monitor traffic. The messaging should be driven by the rate of update loop onboard. The mirror spins at a higher rate, but only publishes on change so it should not be that significant. There is no other message traffic other than basic ROS 2 comms. The new flexbe_webui uses websockets between node and UI window, but the flexbe_app is just internal java script code wrapping some python code to send ROS messages. No outgoing traffic to external web is intended.

[MA16M]

We also experienced the same issue before. Our scenario was robot running on WiFi, PC connected to the same network with LAN. DDS config was the default for iron with multicast discovery. The basic Hello world behaviors were working, but starting up the flexbe_ocs, after more states and behaviors were loaded, the system froze the network immediately. After 2-5minutes the app started to show Online and the network stabilized. After that we could start some behaviors. We also couldn't solve the issue. After swapping to fastdds discovery server the issue disappeared by itself and the network was stable.
The data what we measured:
default dds on launch: 850Mbit/s
discovery server: 39Mbit/s

[RPRKLR]

When I start the OCS only, it still freezes for 2-5 minutes, the network load is around 200Mbit/s.
We also started to investigate the flexbe_webui, there we do not encounter the freezing issue, but we encounter other issues. If I am not able to solve the issues then I am going to open an other issue for the webui repository.

[dcconner]

@RPRKLR Please file issue tickets for anything related to the flexbe_webui there, or contact me directly https://flexbe.readthedocs.io/en/latest/contactinfo.html with more general questions. I am looking for results from testing the WebUI "in the wild".

We have also seen issues related to multicast discovery on our network unrelated to FlexBE, so you might try the fastdds discovery server setup as mentioned in comment from MA15M above.