Troubleshooting

Testing and Troubleshooting

It is not uncommon to experience problems with delivering network traffic from the source to the destinations when using E2SAR with a load balancer.

Some common problems include:

• Firewalls
• Incorrect routing
• Improper MTU setting
• IPv4 vs IPv6 connectivity issues

This document presents a few troubleshooting procedures to help rule out simple network or configuration issues.

Testing load balancer control plane reachability

If provided with an admin EJFAT_URI (in the form of ejfats://<admin token>@<control plane host>:<control plane port>/), you should test the reachability of the control plane host and port as follows:

• ping control plane host name
  • if unsuccessful use dig command to determine host IP address and try to ping the address directly
  • Sometimes a host may resolve to IPv4 and IPv6 address, determine which of the two is reachable via ping and see the next section for suggestions
• once the ping is successful you can use nc command to try to reach the control plane port - generally the command would silently hang if the port is reachable
• E2SAR starting from version 0.2.2 includes a helpful test-tcp-connectivity.sh script that performs these steps in order and reports the results
• use lbadm command to run --version command on the control plane and make sure it returns a non-error response: lbadm --version -u 'ejfats://<admin token>@<control plane host>:<control plane port>/'
  • if you encounter TLS/SSL issues try adding -v flag to the lbadm command line - this disables SSL certificate validation

IPv4 vs IPv6 issues

In modern environments it is not uncommon to see two valid networking configurations - IPv4 and IPv6. E2SAR provides you with controls on which you want to use. You can

• Replace host name with an appropriate IP address in the admin or instance EJFAT_URI
• Use preferV6 flag in EjfatURI constructor if using IPv6 is preferred over v6 when using the control plane hostname to reach the load balancer
• Use SegmenterFlags.dpV6 flag to select which data= address (v4 or v6) is to be used to send data to the load balancer. These are returned as part of the instance Ejfat URI as e.g. ejfats://<instance token>@<control plane host>:<control plane port>/lb/20?sync=129.57.177.130:19532&data=129.57.177.66&data=[2620:0:22f0:677::177:42]
• Use v6 flag on Reassembler constructor to favor IPv6 address for receiving data, when using IP address auto-detection (not specifying IP address explicitly)
• Note it is currently not possible to bridge IPv4 and IPv6 networks through the load balancer. All senders and receivers must be using either the same address family (IPv4 or IPv6).

Testing the dataplane reachability

If you have an instance EJFAT URI in the form of ejfats://<instance token>@<control plane host>:<control plane port>/lb/20?sync=129.57.177.130:19532&data=129.57.177.66&data=[2620:0:22f0:677::177:42] which has both an IPv4 data= address and an IPv6 data= address of the load balancer. First determine whether you will be using IPv4 or IPv6 to send and receive your data. Then from every sender and receiver node

• Determine basic reachability by using ping to the appropriate data= address
• If you have multiple interfaces, narrow down which interface/IP address will be used by using ping with -I option
• Determine that the path to the load balancer has an appropriate MTU by using ping as follows:
  • ping -M do -s 8972 -c 5 <selected data= address> if you expect MTU of 9000 to work
  • ping -M do -s 1472 -c 5 <selected data= address> if you expect MTU of 1500 to work
• Notice that the same MTU must work from sender to load balancer and from load balancer to all receivers. If a larger MTU doesn't work, try making it smaller.
• E2SAR starting from version 0.2.2 includes a helpful test-udp-connectivity.sh script that performs many of these steps and reports the results

Determine which interface and IP address will be used for the dataplane

Both E2SAR Segmenter and Reassembler can auto-detect the outgoing interface and address, however for your debugging you must have that information yourself. Use ip r get command to achieve that:

$ ip route get <selected data= address>

The response if successful will provide you both the interface name and the address to be used if you want to override the auto-detected selection for both Segmenter and Reassembler.

Testing the basic workflow

If you have E2SAR installed on your system, bash scripts installed under /usr/local/bin or found in the source tree under scripts/bash-helpers/ can help validate the basic workflow - running the scripts with appropriate parameters in the order described in the README.md file helps validate the various pieces:

• Querying the load balancer
• Reserving a load balancer instance
• Checking the status of the instance
• Sending and receiving event data over the dataplane through the load balancer
• Freeing load balancer instance

Alternatively you can run the executables (without the Bash scripts) using Docker, without installing any packages. General invocation is:

$ docker run --rm ibaldin/e2sar lbadm -h

or

$ docker run --rm --network host ibaldin/e2sar e2sar_perf -h

Note that you must use host networking mode in order to use e2sar_perf so it can use native host network interfaces. lbadm can be run without it.