Networking plugin: failed to create veth pair ('veth0' : 'eth0') (12 - Object not found)

[anttikyl]

Description

Using a Raspberry Pi 3 RDK build, the networking plugin fails to create a veth pair on container launch.

Visible with network types nat or none.

Workaround is to only use open network type to circumvent having to create a veth device.

Expected Behaviour

Container launches fine with no errors, creating a network of type none or nat.

Actual Behaviour

The logs show:

0000000571.171655 ERR: < M:Netlink.cpp F:createVeth L:1167 > failed to create veth pair ('veth0' : 'eth0') (12 - Object not found)
0000000571.171704 ERR: < M:NetworkSetup.cpp F:setupVeth L:715 > failed to create veth pair for container 'asd'
0000000571.171820 ERR: < M:NetworkingPlugin.cpp F:createRuntime L:209 > failed to setup virtual ethernet device
0000000571.171913 WRN: < M:DobbyRdkPluginManager.cpp F:runPlugins L:630 > Non-required plugin networking createRuntime hook has failed. Continuing running other plugins.

Container starts, but networking isn't working as expected, because the plugin didn't finalise setup.

Steps to Reproduce

Start any container with networking plugin type set as none or nat on the Raspberry Pi 3 Platco build.

Platform/Environment

• Raspberry Pi 3 Model B v1.2
• rdk-firebolt-mediaclient-image image, raspberrypi-cpc-mc machine, 2006_sprint branch.
• Crun version: 0.14.1
• Dobby version: 3.0 (master branch, commit 58ca105)

[anttikyl]

The error Object not found is seen as -NLE_OBJ_NOTFOUND in libnl. The function that returns that error is rtnl_link_veth_add, which should not return that error.

I can replicate the issue on the Vagrant VM by passing a garbage pid to the function.

I've checked with a custom build of the networking plugin on the RPI to confirm that the pid passed to the function is the correct one, and I can confirm that it is.

I then followed this guide to use strace and pyroute2 to check the system calls and decrypt netlink messages.

The output of sendmsg at the point where libnl sends a message to netlink is the following:

RPI strace (sendmsg) -> pyroute2 -> decrypt output:

68:00:00:00:10:00:05:06:8f:8e:6c:5f:e7:35:40:74:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:0a:00:03:00:76:65:74:68:31:00:00:00:3c:00:12:00:09:00:01:00:76:65:74:68:00:00:00:00:2c:00:02:00:28:00:01:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:09:00:03:00:65:74:68:30:00:00:00:00:08:00:13:00:e5:35:00:00
{'attrs': [('RTA_UNSPEC', None),
           ('RTA_IIF', 1752458614),
           ('RTA_VIA', {'addr': '01:00:76:65:74:68:00:00:00:00:2c:00:02:00:28:00:01:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:09:00:03:00:65:74:68:30:00:00:00:00:08:00:13:00:e5:35:00:00', 'family': 9})],
 'dst_len': 0,
 'family': 0,
 'flags': 0,
 'header': {'flags': 1541,
            'length': 104,
            'pid': 1950365159,
            'sequence_number': 1600949903,
            'type': 16},
 'proto': 0,
 'scope': 0,
 'src_len': 0,
 'table': 0,
 'tos': 0,
 'type': 0}

Then looking at the same exact message on the Xi1 2006_sprint build:

RPI strace (sendmsg) -> pyroute2 -> decrypt output:

68:00:00:00:10:00:05:06:e4:5b:6c:5f:02:00:c0:f9:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:0a:00:03:00:76:65:74:68:30:00:00:00:3c:00:12:00:09:00:01:00:76:65:74:68:00:00:00:00:2c:00:02:00:28:00:01:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:09:00:03:00:65:74:68:30:00:00:00:00:08:00:13:00
{'attrs': [('RTA_UNSPEC', None),
           ('RTA_IIF', 1752458614),
           ('RTA_VIA', '<\x00\x12\x00\t\x00\x01\x00veth\x00\x00\x00\x00,\x00\x02\x00(\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\t\x00\x03\x00eth0\x00\x00\x00\x00\x08\x00\x13\x00')],
 'dst_len': 0,
 'family': 0,
 'flags': 0,
 'header': {'flags': 1541,
            'length': 104,
            'pid': 4190109698,
            'sequence_number': 1600936932,
            'type': 16},
 'proto': 0,
 'scope': 0,
 'src_len': 0,
 'table': 0,
 'tos': 0,
 'type': 0}

The RTA_VIA message content is different, and doesn't look to be matching what pyroute2 is expecting. The message contains veth0 and eth0 strings in hex in both of them.

Looking further into the contents, it looks like the Xi1's message contains the expected pid (0x35, 0xe5 = 13797), but the message of the RPi does not (expecting 13259 = 0xcb, 0x33). The message content of the RPi is also 4 bytes shorter than the Xi1.

This to me indicates that the pid is not being sent in the message correctly on the RPi, causing the -NLE_OBJ_NOTFOUND error.