ERROR: Creating the container failed. Error was unrecoverable

[Serrvosky]

Hello guys,

I'm trying to set up my nomad to work with LXC, but I'm getting an error with no useful information.

When I'm deploying my LXC job,I'm getting the following errors in my nomad client:

Logs

Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.619Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example path=/etc/nomad.d/alloc/5f59f5a4-7ef9-b462-afc1-b7407810ae53/alloc/logs/.example.stdout.fifo @module=logmon timestamp=2019-11-12T09:03:17.619Z
Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.620Z [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example @module=logmon path=/etc/nomad.d/alloc/5f59f5a4-7ef9-b462-afc1-b7407810ae53/alloc/logs/.example.stderr.fifo timestamp=2019-11-12T09:03:17.619Z
Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.671Z [INFO ] client.driver_mgr.nomad-driver-lxc: starting lxc task: driver=lxc @module=lxc driver_cfg="{Template:/usr/share/lxc/templates/lxc-busybox Distro: Release: Arch: ImageVariant: ImageServer: GPGKeyID: GPGKeyServer: DisableGPGValidation:false FlushCache:false ForceCache:false TemplateArgs:[] LogLevel:trace Verbosity:verbose Volumes:[]}" timestamp=2019-11-12T09:03:17.670Z
Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.676Z [ERROR] client.alloc_runner.task_runner: rnning driver failed: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example error="rpc error: code = Unknown desc = unable to create container: creating the container failed"
Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.676Z [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example reason="Error was unrecoverable"
Nov 12 09:03:17 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:17.686Z [INFO ] client.gc: marking allocation for GC: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53
Nov 12 09:03:21 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:21.688Z [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example @module=logmon timestamp=2019-11-12T09:03:21.687Z
Nov 12 09:03:21 SERVICES-SRV-01 nomad[17452]:     2019-11-12T09:03:21.688Z [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=5f59f5a4-7ef9-b462-afc1-b7407810ae53 task=example @module=logmon timestamp=2019-11-12T09:03:21.688Z

As you can see, it's not easy to debug, because there are no explicit log.

At the moment, this are my environment versions:

Nomad: Nomad v0.9.5
Client: Debian Buster
Kernel Version: 4.9.0-11-amd64
Nomad LXC Plugin: nomad-driver-lxc_0.1.0-rc2 (also tried with nomad-driver-lxc_0.1.0)
LXC Version: v1:3.1.0+really3.0.3-8.
lxc-templates: v3.0

Job Definition
At this moment, i'm trying to deploy this example job, from hashicorp repo:
Job Template - https://github.com/hashicorp/nomad-education-content/blob/master/lxc.nomad

Client Config
This is my nomad client config:

datacenter = "<dc1>"
data_dir = "/etc/nomad.d"
plugin_dir = "/opt/nomad/data/plugins"

bind_addr = "<ip>"

addresses {
  http = "<ip>"
  rpc  = "<ip>"
  serf = "<ip>"
}

advertise {
  # Defaults to the node's hostname. If the hostname resolves to a loopback
  # address you must manually configure advertise addresses.
  http = "<ip>:4646"
  rpc  = "<ip>:4647"
  serf = "<ip>:4648"
}

client {
  network_interface = "<interface>"
  enabled = true
  options {
    "driver.raw_exec.enable" = "1"
  }
}

plugin "nomad-driver-lxc" {
  config {
    enabled = true
    volumes_enabled = true
    lxc_path = "/var/lib/lxc"
  }
}

[towe75]

Hi,

maybe debug logging reveals more?

I assume you already try to create a busybox container manually to verify correct lxc setup?