[libnetwork] Default Profile for a new IPPool generates ipset with no members. Ingress packet to a workload always matches no profile and gets dropped by iptables.

[Syraxius]

Expected Behavior

The default profile generated should allow all egress traffic, and allow ingress traffic from the same network as specified in the documentation page here.

Current Behavior

All egress traffic are permitted, but no ingress traffic is permitted other than from the node itself to its own workload. This is caused by an empty ipset which is supposed to match the IPPool's subnet.

In summary:

• Working: Workload to external
• Working: Workload to node
• Not Working: Workload to workload (dropped at node iptables)
• Not Working: External to workload (dropped at node iptables)

Possible Solution

The issue I was having was fixed when changing the Profile from:

source:
      selector: has(calico-pool-0)

To:

source: {}

(The name of my IPPool resource and Profile are both calico-pool-0)

Of course, the above will now allow everything which is not in the spirit of the default profile.

I will find time soon to determine from code the actual reason why has(calico-pool-0) causes an empty ipset.

Steps to Reproduce (for bugs)

1. Create a new IPPool with this configuration:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: calico-pool-0
spec:
  cidr: 192.168.101.0/24
  blockSize: 26
  ipipMode: CrossSubnet
  natOutgoing: true
  disabled: false

2. Add a new network in Docker:

docker network create --driver calico --ipam-driver calico-ipam --subnet=192.168.101.0/24 calico-pool-1

3. Create two workloads on different hosts:

# Host 1
docker run --net=calico-pool-0 --name w1 -dit busybox sh
# Host 2
docker run --net=calico-pool-0 --name w2 -dit busybox sh

4. Try to ping w2 from w1. Doesn't work. Details are as follows...

Here is the tcpdump (from tcpdump -enni any icmp):

15:13:05.704290  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 12, length 64
15:13:06.704784  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 13, length 64
15:13:07.705399  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 14, length 64
15:13:08.705967  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 15, length 64
15:13:09.706307  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 16, length 64
15:13:10.706950  In 08:00:27:cb:0e:5c ethertype IPv4 (0x0800), length 100: 192.168.101.1 > 192.168.101.193: ICMP echo request, id 3072, seq 17, length 64

It seems it's getting dropped by iptables because it's not getting marked in cali-pro-calico-pool-0 chain:

Chain cali-pro-calico-pool-0 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  any    any     anywhere             anywhere             /* cali:faAFuoL5iNMQqlTy */ MARK or 0x10000
    0     0 RETURN     all  --  any    any     anywhere             anywhere             /* cali:mwhwnGS9ylSlHDLE */ mark match 0x10000/0x10000

Chain cali-tw-calif42bbb120f0 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere             /* cali:2e1wtQwhc6xY4lxs */ ctstate RELATED,ESTABLISHED
    0     0 DROP       all  --  any    any     anywhere             anywhere             /* cali:-TWo_rIkIp3cS0fT */ ctstate INVALID
   76  6384 MARK       all  --  any    any     anywhere             anywhere             /* cali:yJvGboVd7BQ96sDz */ MARK and 0xfffeffff
   76  6384 cali-pri-calico-pool-0  all  --  any    any     anywhere             anywhere             /* cali:hLdAvuX7r-VmswBt */
    0     0 RETURN     all  --  any    any     anywhere             anywhere             /* cali:E4J1GHCRA1rLqJte */ /* Return if profile accepted */ mark match 0x10000/0x10000
   76  6384 DROP       all  --  any    any     anywhere             anywhere             /* cali:Xl8oR2dSw6Awy0oF */ /* Drop if no profiles matched */

From iptables-save -c we can see that the ipset (cali40s:Gcr-rptUOZhd-mtexAmjwZt) is used to match the packets to be marked:

[0:0] -A cali-pri-calico-pool-0 -m comment --comment "cali:asFUIp0qsmlr9w1E" -m set --match-set cali40s:Gcr-rptUOZhd-mtexAmjwZt src -j MARK --set-xmark 0x10000/0x10000
[0:0] -A cali-pri-calico-pool-0 -m comment --comment "cali:8gezXKKJqj4pVICf" -m mark --mark 0x10000/0x10000 -j RETURN

And it seems like the ipset cali40s:Gcr-rptUOZhd-mtexAmjwZt is empty:

Name: cali40s:Gcr-rptUOZhd-mtexAmjwZt
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 1048576
Size in memory: 384
References: 1
Members:

I've tried enabling --use-docker-networking-container-labels but it still does not work.

Context

Trying to run Calico 3.4 with plain Docker. I've compiled libnetwork-plugin and added it to the calico/node image.

Your Environment

• Felix version (if using Felix binary directly calico-felix --version):flag: v3.4.0 (from calico-node -v)
• And/Or, calico/node container version (if running Felix in a container): release-v3.4
• Which orchestrator are you using (e.g. Kubernetes, OpenStack, Docker, Mesos): Docker
• Orchestrator version (if used): N/A
• Etcd version (if using etcd datastore driver): 3.3.8
• Operating System and version: Ubuntu 16.04 amd64
• Link to your project (optional): N/A

[caseydavenport]

source:
      selector: has(calico-pool-0)

I don't think this sort of thing works out of the box - rule selectors match workload endpoints, not IP pools. So, unless your workload endpoints or NetworkSets are labeled with that particular label then that rule won't select anything.

You can check what labels are applied to your workload endpoints with calicoctl get workloadendpoints -o wide --all-namespaces

[Syraxius]

Thanks for your prompt reply @caseydavenport !

Actually what I meant was... The selector configuration was automatically generated (I didn't create it) when the docker network and test containers were created, and it seems to lead to iptables rules / ipsets which drop all traffic (iptables uses an ipset which is empty).

When I run the calicoctl get workloadendpoints -o yml --all-namespaces command, the calico-pool-0 label is not applied to the workload. My workloads only have the following labels:

projectcalico.org/namespace: libnetwork
projectcalico.org/orchestrator: libnetwork

It seems the default selector expects the name of the network (calico-pool-0) to be attached as a label to containers created in that network. However, the label is missing which leads to all traffic being dropped.

Thanks again!

[caseydavenport]

@Syraxius Ah, right I see what you mean.

docker network create --driver calico --ipam-driver calico-ipam --subnet=192.168.101.0/24 calico-pool-1

I noticed in your example command uses calico-pool-1 - I suspect this isn't relevant, but thought I'd call it out anyway in case it was more than just a typo.

This seems to be the only code which is setting labels currently: https://github.com/projectcalico/libnetwork-plugin/blob/master/driver/network_driver.go#L648-L667

From a quick read, I don't see it setting the network name as a label, which I would assume is the root cause here.

As you pointed out, the profile expects a networkName label, and that aligns with the code here: https://github.com/projectcalico/libnetwork-plugin/blob/master/driver/network_driver.go#L361

A simple fix, I think, would be to add the label at this point: https://github.com/projectcalico/libnetwork-plugin/blob/master/driver/network_driver.go#L300