elasticsearch on swarm cluster, ulimits ignored #88
Description
- This is a bug report
- This is a feature request
- I searched existing issues before opening this one
Expected behavior
i'm trying to deploy an elastic service across swarm cluster using compose file.
docker-compose starts elastic container normally, applying ulimits settings from .yml file.
but docker stack deploy ignores ulimits directive, so elasticsearch doesn't start at all.
.yml file is always the same, ether it docker-compose deploy or docker stack deploy
xx@xx:~/git/elk-compose-full$ docker-compose -f elastic-compose-single.yml up
WARNING: Some services (elasticsearch) use the 'deploy' key, which will be ignored. Compose does not support 'deploy' configuration - use `docker stack deploy` to deploy to a swarm.
WARNING: The Docker Engine you're using is running in swarm mode.
Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.
To deploy your application across the swarm, use `docker stack deploy`.
Creating network "elkcomposefull_esnet" with the default driver
Creating volume "elkcomposefull_data" with local driver
Creating elasticsearch ...
Creating elasticsearch ... done
Attaching to elasticsearch
elasticsearch | [2017-08-29T09:15:52,331][INFO ][o.e.n.Node ] [] initializing ...
elasticsearch | [2017-08-29T09:15:52,397][INFO ][o.e.e.NodeEnvironment ] [TibRwks] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/tmpl--vg-root)]], net usable_space [2.8gb], net total_space [7.2gb], spins? [possibly], types [ext4]
elasticsearch | [2017-08-29T09:15:52,397][INFO ][o.e.e.NodeEnvironment ] [TibRwks] heap size [494.9mb], compressed ordinary object pointers [true]
elasticsearch | [2017-08-29T09:15:52,398][INFO ][o.e.n.Node ] node name [TibRwks] derived from node ID [TibRwkssREK1G3d_32IsXg]; set [node.name] to override
elasticsearch | [2017-08-29T09:15:52,399][INFO ][o.e.n.Node ] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-92-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elasticsearch | [2017-08-29T09:15:52,399][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [aggs-matrix-stats]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [ingest-common]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [lang-expression]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [lang-groovy]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [lang-mustache]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [lang-painless]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [parent-join]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [percolator]
elasticsearch | [2017-08-29T09:15:53,815][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [reindex]
elasticsearch | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [transport-netty3]
elasticsearch | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService ] [TibRwks] loaded module [transport-netty4]
elasticsearch | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService ] [TibRwks] loaded plugin [ingest-geoip]
elasticsearch | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService ] [TibRwks] loaded plugin [ingest-user-agent]
elasticsearch | [2017-08-29T09:15:53,816][INFO ][o.e.p.PluginsService ] [TibRwks] loaded plugin [x-pack]
elasticsearch | [2017-08-29T09:15:55,638][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.2 (Build 0d83940bb9d682) Copyright (c) 2017 Elasticsearch BV
elasticsearch | [2017-08-29T09:15:55,666][INFO ][o.e.d.DiscoveryModule ] [TibRwks] using discovery type [zen]
elasticsearch | [2017-08-29T09:15:56,261][INFO ][o.e.n.Node ] initialized
elasticsearch | [2017-08-29T09:15:56,261][INFO ][o.e.n.Node ] [TibRwks] starting ...
elasticsearch | [2017-08-29T09:15:56,452][INFO ][o.e.t.TransportService ] [TibRwks] publish_address {172.19.0.2:9300}, bound_addresses {0.0.0.0:9300}
elasticsearch | [2017-08-29T09:15:56,466][INFO ][o.e.b.BootstrapChecks ] [TibRwks] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elasticsearch | [2017-08-29T09:15:59,643][INFO ][o.e.c.s.ClusterService ] [TibRwks] new_master {TibRwks}{TibRwkssREK1G3d_32IsXg}{ZAdw2rV1Q2-5ZuiP7PIhvg}{172.19.0.2}{172.19.0.2:9300}{ml.max_open_jobs=10, ml.enabled=true}, reason: zen-disco-elected-as-master ([0] nodes joined)
elasticsearch | [2017-08-29T09:15:59,716][INFO ][o.e.h.n.Netty4HttpServerTransport] [TibRwks] publish_address {172.19.0.2:9200}, bound_addresses {0.0.0.0:9200}
elasticsearch | [2017-08-29T09:15:59,716][INFO ][o.e.n.Node ] [TibRwks] started
elasticsearch | [2017-08-29T09:15:59,764][INFO ][o.e.g.GatewayService ] [TibRwks] recovered [0] indices into cluster_state
elasticsearch | [2017-08-29T09:15:59,978][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-state index template
elasticsearch | [2017-08-29T09:15:59,990][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-meta index template
elasticsearch | [2017-08-29T09:16:00,026][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-anomalies- index template
elasticsearch | [2017-08-29T09:16:00,039][INFO ][o.e.x.m.MachineLearningTemplateRegistry] [TibRwks] successfully created .ml-notifications index template
elasticsearch | [2017-08-29T09:16:00,214][INFO ][o.e.l.LicenseService ] [TibRwks] license [e1688e1c-cd69-49f4-9e39-fd2a621f2558] mode [trial] - valid
Actual behavior
docker stack deploy -c elastic-compose-single.yml elk
xx@xx:~/git/elk-compose-full$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
q0u1pnz6zc2k elk_elasticsearch replicated 1/1 docker.elastic.co/elasticsearch/elasticsearch:5.5.2 *:9200->9200/tcp
xx@xx:~/git/elk-compose-full$ docker service logs q0u1pnz6zc2k
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:52,929][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
elk_elasticsearch.1.a83hfcmmfma7@xx | # allow user 'elasticsearch' mlockall
elk_elasticsearch.1.a83hfcmmfma7@xx | elasticsearch soft memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xx | elasticsearch hard memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,042][INFO ][o.e.n.Node ] [] initializing ...
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,153][INFO ][o.e.e.NodeEnvironment ] [dT7I27E] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/tmpl--vg-root)]], net usable_space [2.8gb], net total_space [7.2gb], spins? [possibly], types [ext4]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,153][INFO ][o.e.e.NodeEnvironment ] [dT7I27E] heap size [494.9mb], compressed ordinary object pointers [true]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node ] node name [dT7I27E] derived from node ID [dT7I27EwQOaQpekZF0mVsw]; set [node.name] to override
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node ] version[5.5.2], pid[1], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-92-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_141/25.141-b16]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:53,155][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [aggs-matrix-stats]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [ingest-common]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [lang-expression]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [lang-groovy]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,608][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [lang-mustache]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [lang-painless]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [parent-join]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,609][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [percolator]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [reindex]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [transport-netty3]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded module [transport-netty4]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded plugin [ingest-geoip]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,610][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded plugin [ingest-user-agent]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:54,611][INFO ][o.e.p.PluginsService ] [dT7I27E] loaded plugin [x-pack]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:56,448][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/55] [Main.cc@128] controller (64 bit): Version 5.5.2 (Build 0d83940bb9d682) Copyright (c) 2017 Elasticsearch BV
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:56,479][INFO ][o.e.d.DiscoveryModule ] [dT7I27E] using discovery type [zen]
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,116][INFO ][o.e.n.Node ] initialized
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,116][INFO ][o.e.n.Node ] [dT7I27E] starting ...
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,314][INFO ][o.e.t.TransportService ] [dT7I27E] publish_address {10.0.0.2:9300}, bound_addresses {0.0.0.0:9300}
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,325][INFO ][o.e.b.BootstrapChecks ] [dT7I27E] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
elk_elasticsearch.1.a83hfcmmfma7@xx | ERROR: [1] bootstrap checks failed
elk_elasticsearch.1.a83hfcmmfma7@xx | [1]: memory locking requested for elasticsearch process but memory is not locked
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,333][INFO ][o.e.n.Node ] [dT7I27E] stopping ...
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,384][INFO ][o.e.n.Node ] [dT7I27E] stopped
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,384][INFO ][o.e.n.Node ] [dT7I27E] closing ...
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,408][INFO ][o.e.n.Node ] [dT7I27E] closed
elk_elasticsearch.1.a83hfcmmfma7@xx | [2017-08-29T09:08:57,412][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
Steps to reproduce the behavior
elastic-compose-single.yml:
version: '3'
services:
elasticsearch:
container_name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2
volumes:
- /etc/localtime:/etc/localtime:ro
- data:/usr/share/elasticsearch/data
environment:
- xpack.security.enabled=false
- xpack.monitoring.enabled=false
- cluster.name=elastic-cluster
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ports:
- 9200:9200
ulimits:
memlock:
soft: -1
hard: -1
deploy:
resources:
limits:
memory: 1000M
networks:
- esnet
volumes:
data:
driver: local
networks:
esnet:
xx@xx:~/git/elk-compose-full$ docker stack deploy -c elastic-compose-single.yml elk
Ignoring deprecated options:
container_name: Setting the container name is not supported.
Creating network elk_esnet
Creating service elk_elasticsearch
xx@xx:~/git/elk-compose-full$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
q0u1pnz6zc2k elk_elasticsearch replicated 1/1 docker.elastic.co/elasticsearch/elasticsearch:5.5.2 *:9200->9200/tcp
xx@xx:~/git/elk-compose-full$ docker service logs q0u1pnz6zc2k
elk_elasticsearch.1.a83hfcmmfma7@xxx | [2017-08-29T09:08:52,929][WARN ][o.e.b.JNANatives ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
elk_elasticsearch.1.a83hfcmmfma7@xxx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] This can result in part of the JVM being swapped out.
elk_elasticsearch.1.a83hfcmmfma7@xxx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
elk_elasticsearch.1.a83hfcmmfma7@xxx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] These can be adjusted by modifying /etc/security/limits.conf, for example:
elk_elasticsearch.1.a83hfcmmfma7@xxx | # allow user 'elasticsearch' mlockall
elk_elasticsearch.1.a83hfcmmfma7@xxx | elasticsearch soft memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xxx | elasticsearch hard memlock unlimited
elk_elasticsearch.1.a83hfcmmfma7@xxx | [2017-08-29T09:08:52,931][WARN ][o.e.b.JNANatives ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
Output of docker version:
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false
Output of docker info:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 2
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 11
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: active
NodeID: lrsfbttb3qsasbb04fhhawru3
Is Manager: true
ClusterID: l14zv2dzph783ccusuwg622sp
Managers: 1
Nodes: 4
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Node Address: 192.168.248.40
Manager Addresses:
192.168.248.40:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-92-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 1.953GiB
Name: xxx
ID: FZCA:5HIY:FAYG:JV3Y:Q3NG:M327:GKCD:TZX4:WLX3:MDKI:HX5Z:CRHV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Additional environment details (AWS, VirtualBox, physical, etc.)
docker host on vmware guest esxi 6