API server reports storage error

[langston-barrett]

The API server is logging this error:

I0718 12:13:14.820477       1 consul.go:241] Storage Error: <nil>, key: registry/services/endpoints/default/kubernetes
I0718 12:13:17.931427       1 consul.go:241] Storage Error: <nil>, key: registry/minions/lb-kubeworker-001
I0718 12:13:19.728024       1 consul.go:241] Storage Error: <nil>, key: registry/minions/lb-kubeworker-002
W0718 12:13:24.830956       1 controller.go:312] Resetting endpoints for master service "kubernetes" to kind:"" apiVersion:"" 
I0718 12:13:24.893052       1 consul.go:241] Storage Error: <nil>, key: registry/services/endpoints/default/kubernetes
I0718 12:13:27.968906       1 consul.go:241] Storage Error: <nil>, key: registry/minions/lb-kubeworker-001
I0718 12:13:29.757710       1 consul.go:241] Storage Error: <nil>, key: registry/minions/lb-kubeworker-002

When I use consul-cli to check out that key, it works:

[root@lb-control-01 ~]# consul-cli kv-read registry/minions/lb-kubeworker-002
{"kind":"Node","apiVersion":"v1","metadata":{"name":"lb-kubeworker-002","selfLink":"/api/v1/nodes/lb-kubeworker-002","uid":"a746cc9b-4cda-11e6-8752-04012f09f701","creationTimestamp":"2016-07-18T11:27:47Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/hostname":"lb-kubeworker-002"}},"spec":{"externalID":"lb-kubeworker-002"},"status":{"capacity":{"alpha.kubernetes.io/nvidia-gpu":"0","cpu":"2","memory":"3882448Ki","pods":"110"},"allocatable":{"alpha.kubernetes.io/nvidia-gpu":"0","cpu":"2","memory":"3882448Ki","pods":"110"},"conditions":[{"type":"OutOfDisk","status":"False","lastHeartbeatTime":"2016-07-18T12:13:39Z","lastTransitionTime":"2016-07-18T11:27:47Z","reason":"KubeletHasSufficientDisk","message":"kubelet has sufficient disk space available"},{"type":"Ready","status":"True","lastHeartbeatTime":"2016-07-18T12:13:39Z","lastTransitionTime":"2016-07-18T11:27:47Z","reason":"KubeletReady","message":"kubelet is posting ready status"}],"addresses":[{"type":"LegacyHostIP","address":"139.59.159.78"},{"type":"InternalIP","address":"139.59.159.78"}],"daemonEndpoints":{"kubeletEndpoint":{"Port":10250}},"nodeInfo":{"machineID":"e3715d5076f54310a857957070af9f0b","systemUUID":"E3715D50-76F5-4310-A857-957070AF9F0B","bootID":"32c225f8-1985-4ebf-9e97-72b20e9bb497","kernelVersion":"3.10.0-327.22.2.el7.x86_64","osImage":"CentOS Linux 7 (Core)","containerRuntimeVersion":"docker://1.11.2","kubeletVersion":"v1.3.0-alpha.1.1977+a1e79e91b9fb6a","kubeProxyVersion":"v1.3.0-alpha.1.1977+a1e79e91b9fb6a","operatingSystem":"linux","architecture":"amd64"},"images":[{"names":["gcr.io/google_containers/pause-amd64:3.0"],"sizeBytes":746888},{"names":["ciscocloud/nginx-consul:2.0"],"sizeBytes":23773593},{"names":["calico/node:v0.19.0"],"sizeBytes":91984482},{"names":["gcr.io/google_containers/kubernetes-dashboard-amd64:v1.0.1"],"sizeBytes":44090031},{"names":["skynetservices/skydns:2.5.3a"],"sizeBytes":18864800}]}}

What's going on?

[mikejihbe]

Consul HEAD supports transactions, and since the last mantl release, we have changed our code to utilize that. The next mantl release should include that build instead of our fork and we believe that will resolve this issue.