LXC driver failure: error setting network type configuration: setting config item for the container failed

[yum-install-brains]

Nomad version

Nomad v0.9.6 (1f8eddf2211d064b150f141c86e30d9fceabec89)

Operating system and Environment details

OS name and verison:

Distributor ID:	Debian
Description:	Debian GNU/Linux 9.9 (stretch)
Release:	9.9
Codename:	stretch

Kerner version:

4.19.0-0.bpo.2-amd64

LXC version

ii  liblxc1                          1:3.1.0+really3.0.4-2          amd64        Linux Containers userspace tools (library)
ii  libpam-cgfs                      1:3.1.0+really3.0.4-2          amd64        PAM module for managing cgroups for LXC
ii  lxc                              1:3.1.0+really3.0.4-2          amd64        Linux Containers userspace tools
ii  lxcfs                            3.0.4-2                        amd64        FUSE based filesystem for LXC

Issue

lxc-driver throws an error: Driver Failure rpc error: code = Unknown desc = error setting network type configuration: setting config item for the container failed

Reproduction steps

1. Install nomad

curl -O https://releases.hashicorp.com/nomad/0.9.6/nomad_0.9.6_linux_amd64.zip
unzip nomad_0.9.6_linux_amd64.zip
sudo mv nomad /usr/local/bin
rm nomad_0.9.6_linux_amd64.zip

2. Install lxc driver

sudo apt install -y lxc
sudo mkdir -p /opt/nomad/data/plugins
curl -O https://releases.hashicorp.com/nomad-driver-lxc/0.1.0-rc2/nomad-driver-lxc_0.1.0-rc2_linux_amd64.zip
unzip nomad-driver-lxc_0.1.0-rc2_linux_amd64.zip
sudo mv nomad-driver-lxc /opt/nomad/data/plugins
rm ./nomad-driver-lxc*.zip

3. Define server.hcl and run sudo nomad agent -config server.hcl

# Increase log verbosity
log_level = "DEBUG"

datacenter = "DC1"

# Setup data dir
data_dir = "/tmp/server1"

# Enable the server
server {
    enabled = true

    # Self-elect, should be 3 or 5 for production
    bootstrap_expect = 1
}

4. Define client.hcl and run sudo nomad agent -config client.hcl

# Increase log verbosity
log_level = "DEBUG"

datacenter = "DC1"

# Setup data dir
data_dir = "/opt/nomad/data"

# Give the agent a unique name. Defaults to hostname
name = "nomad-agent"

# Enable the client
client {
    enabled = true

    # For demo assume we are talking to server1. For production,
    # this should be like "nomad.service.consul:4647" and a system
    # like Consul used for service discovery.
    servers = ["nomad-server:4647"]
}

# Modify our port to avoid a collision with server1
ports {
    http = 5656
}

plugin "nomad-driver-lxc" {
  config {
    enabled = true
    lxc_path = "/var/lib/lxc"
  }
}

5. Create job

job "example-lxc" {
  datacenters = ["DC1"]
  type        = "service"

  group "lxc" {
    task "busybox" {
      driver = "lxc"

      config {
        log_level = "trace"
        verbosity = "verbose"
        template  = "/usr/share/lxc/templates/lxc-busybox"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

6. Run job
   nomad job run -address nomad-server:4646

And it will fail

ID                     = 3e12b0e1
Eval ID                = a0f6d22d
Name                   = example-lxc.lxc4[0]
Node ID                = 28751cda
Node Name              = nomad-agent
Job ID                 = example-lxc
Job Version            = 824638796448
Client Status          = failed
Client Description     = Failed tasks
Desired Status         = run
Desired Description    = <none>
Created                = 12s ago
Modified               = 8s ago
Reschedule Eligibility = 18s from now

Task "busybox" is "dead"
Task Resources
CPU      Memory   Disk     Addresses
500 MHz  256 MiB  300 MiB

Task Events:
Started At     = N/A
Finished At    = 2019-10-14T15:30:59Z
Total Restarts = 0
Last Restart   = N/A

Recent Events:
Time                       Type            Description
2019-10-14T18:30:59+03:00  Killing         Sent interrupt. Waiting 5s before force killing
2019-10-14T18:30:59+03:00  Not Restarting  Error was unrecoverable
2019-10-14T18:30:59+03:00  Driver Failure  rpc error: code = Unknown desc = error setting network type configuration: setting config item for the container failed
2019-10-14T18:30:58+03:00  Task Setup      Building Task Directory
2019-10-14T18:30:58+03:00  Received        Task received by client

[yum-install-brains]

We did some additional tests and downgraded lxc from 1:3.1.0+really3.0.4-2 to 1:2.1.0-0. With 1:2.1.0-0 all works fine.

Do you have any plans to support LXCv3?

[yum-install-brains]

I changed this function from nomad-driver-lxc to always return "lxc.net.0.type" and somehow it works with lxc 3.0.4 now. Do not know why (mb lxc.VersionAtLeast is not working properly with 3rd version of LXC).

[notnoop]

Hi @yum-install-brains ! Thanks for reaching out. It's definitely a bug that current driver doesn't support LXC 3. In my brief testing, I verified that the published binary fails with the error you reported, but I succeed when I recompile the current nomad-driver-lxc without any modification and use it with nomad. I wonder if there is an issue with the way we link against lxc library.

Can you try using a recompiled version but without your modification to networkTypeConfigKey? We'll investigate on our end too. Thanks!

[notnoop]

@yum-install-brains Thank you so much for reporting this again, it's a tricky one! - I reported the underlying issue to the lxc library we use: lxc/go-lxc#135 and will follow up there.

[notnoop]

At this point, I'd recommend folks to recompile their nomad-driver-lxc with the lxc version they are running against. We'd appreciate pull requests to workaround this issue - or we may address it when the upstream issue is addressed.

[jinnatar]

Would it be possible to provide pre-built binaries at least for the most common LXC versions out there? For example 3.0.3 that ships with Ubuntu 18.04.

I did try to compile the plugin myself but building go is quite a painful experience for someone who isn't a go developer. Your official instructions of make build is not nearly enough when all the dependencies are not pulled in and the usual go get method crashes on sizeofPtr redeclaration errors. I assume I don't have a recent enough version of go, but that then goes into a whole rabbithole of dealing with a non-system Go installation when I'm just attempting to build one wayward plugin. :-)

Edit:
Was able to use a PPA to get golang-go 1.15.6: https://github.com/golang/go/wiki/Ubuntu .. then the build works ok. So might want to add a minimum go version notice in the README. I know go abandons versions on the regular but it's a bit annoying for LTS users.

[h0tw1r3]

Likely related to this bug in go-lxc
lxc/go-lxc#135