Skip to content

FAILED - RETRYING: Register nodes #2

New issue
New issue
Open
Open
FAILED - RETRYING: Register nodes#2
@solomonjsands

Description

@solomonjsands
solomonjsands
opened on Mar 21, 2018

Hello,

I'm deploying this with 4 nodes: 1 kolla controller, 1 contrail controller, 2 compute nodes. I can deploy without error when doing an all-in-one. In this setup I am getting blocked as listed in my notes below. It seems the webui container on kolla2(contrail controller) is not created and thus port 8095 is not available. The openstack-kolla controller and computes are operational, just contrail getting stuck. Please advise

root@kolla1:~/OpenContrail-Kolla/contrail-ansible/playbooks# ansible-playbook -vvv -i inventory/my-inventory site.yml
...
TASK [node : Register nodes, Wait till config api server answers] *****************************************************************
task path: /root/OpenContrail-Kolla/contrail-ansible/playbooks/roles/node/tasks/register_with_controller.yml:2
Using module file /usr/local/lib/python2.7/dist-packages/ansible/modules/commands/command.py
<kolla2.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: root
<kolla2.mydomain.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/304593b28d kolla2.mydomain.com '/bin/sh -c '"'"'echo ~ && sleep 0'"'"''
<kolla2.mydomain.com> (0, '/root\n', '')
<kolla2.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: root
<kolla2.mydomain.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/304593b28d kolla2.mydomain.com '/bin/sh -c '"'"'( umask 77 && mkdir -p "echo /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184" && echo ansible-tmp-1521656406.73-242997469214184="echo /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184" ) && sleep 0'"'"''
<kolla2.mydomain.com> (0, 'ansible-tmp-1521656406.73-242997469214184=/root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184\n', '')
<kolla2.mydomain.com> PUT /tmp/tmphWf91h TO /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/command.py
<kolla2.mydomain.com> SSH: EXEC sftp -b - -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/304593b28d '[kolla2.mydomain.com]'
<kolla2.mydomain.com> (0, 'sftp> put /tmp/tmphWf91h /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/command.py\n', '')
<kolla2.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: root
<kolla2.mydomain.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/304593b28d kolla2.mydomain.com '/bin/sh -c '"'"'chmod u+x /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/ /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/command.py && sleep 0'"'"''
<kolla2.mydomain.com> (0, '', '')
<kolla2.mydomain.com> ESTABLISH SSH CONNECTION FOR USER: root
<kolla2.mydomain.com> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/root/.ssh/id_rsa"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/304593b28d -tt kolla2.mydomain.com '/bin/sh -c '"'"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/command.py; rm -rf "/root/.ansible/tmp/ansible-tmp-1521656406.73-242997469214184/" > /dev/null 2>&1 && sleep 0'"'"''
<kolla2.mydomain.com> (0, '\r\n{"changed": true, "end": "2018-03-21 14:20:09.115256", "stdout": "000", "cmd": "docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095", "failed": true, "delta": "0:00:00.914077", "stderr": "", "rc": 7, "invocation": {"module_args": {"warn": true, "executable": null, "_uses_shell": true, "_raw_params": "docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095", "removes": null, "creates": null, "chdir": null, "stdin": null}}, "start": "2018-03-21 14:20:08.201179", "msg": "non-zero return code"}\r\n', 'Shared connection to kolla2.mydomain.com closed.\r\n')
FAILED - RETRYING: Register nodes, Wait till config api server answers (179 retries left).Result was: {
"attempts": 94,
"changed": true,
"cmd": "docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095",
"delta": "0:00:00.914077",
"end": "2018-03-21 14:20:09.115256",
"invocation": {
"module_args": {
"_raw_params": "docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095",
"_uses_shell": true,
"chdir": null,
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"warn": true
}
},
"msg": "non-zero return code",
"rc": 7,
"retries": 181,
"start": "2018-03-21 14:20:08.201179",
"stderr": "",
"stderr_lines": [],
"stdout": "000",
"stdout_lines": [
"000"
]
}

...
FAILED - RETRYING: Register nodes, Wait till config api server answers (1 retries left).
fatal: [kolla2.mydomain.com]: FAILED! => {"attempts": 180, "changed": true, "cmd": "docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095", "delta": "0:00:00.932515", "end": "2018-03-21 11:55:28.465676", "msg": "non-zero return code", "rc": 7, "start": "2018-03-21 11:55:27.533161", "stderr": "", "stderr_lines": [], "stdout": "000", "stdout_lines": ["000"]}

root@kolla2:# docker exec controller curl -s -o /dev/null --write-out %{http_code} -uadmin:contrail1 http://localhost:8095
000
root@kolla2:# docker exec controller http://localhost:8095
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to localhost port 8095: Connection refused

root@kolla2:~# ss -l4
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 *:bootpc :
udp UNCONN 0 0 172.20.0.64:ntp :
udp UNCONN 0 0 127.0.0.1:ntp :
udp UNCONN 0 0 *:ntp :
tcp LISTEN 0 128 *:ssh :
tcp LISTEN 0 50 *:8888 :
tcp LISTEN 0 128 127.0.0.1:6010 :
tcp LISTEN 0 50 127.0.0.1:61619 :
tcp LISTEN 0 50 *:61620 :

root@kolla2:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
082c7859ff29 gokulpch/contrail-agent-ubuntu16.04:4.0.1.0-12 "/lib/systemd/system…" 3 hours ago Up 3 hours agent
fcd36f70e6bb gokulpch/contrail-analytics-ubuntu16.04:4.0.1.0-12 "/lib/systemd/system…" 3 hours ago Up 3 hours analytics
c76a25adbe70 gokulpch/contrail-analyticsdb-ubuntu16.04:4.0.1.0-12 "/lib/systemd/system…" 3 hours ago Up 3 hours analyticsdb
1c4f7bf1304d gokulpch/contrail-controller-ubuntu16.04:4.0.1.0-12 "/lib/systemd/system…" 3 hours ago Up 3 hours controller

root@kolla2:~# docker logs controller

Welcome to Ubuntu 16.04.2 LTS!

[ OK ] Reached target Local File Systems.
[ OK ] Listening on Journal Audit Socket.
[ OK ] Listening on Journal Socket (/dev/log).
[ OK ] Reached target Paths.
[ OK ] Listening on Journal Socket.
[ OK ] Reached target Swap.
[ OK ] Created slice System Slice.
Starting Create Static Device Nodes in /dev...
Starting Create Volatile Files and Directories...
[ OK ] Reached target Slices.
[ OK ] Started Create Static Device Nodes in /dev.
Starting Journal Service...
[ OK ] Started Create Volatile Files and Directories.
[ OK ] Started Journal Service.
[ OK ] Reached target System Initialization.
[ OK ] Started Daily Cleanup of Temporary Directories.
[ OK ] Reached target Timers.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting LSB: Set the CPU Frequency Scaling governor to "ondemand"...
Starting LSB: centralized coordination service...
Starting /etc/rc.local Compatibility...
Starting Contrail controller configuration...
Starting Cleanup of Temporary Directories...
[ OK ] Started /etc/rc.local Compatibility.
[ OK ] Started Cleanup of Temporary Directories.
[ OK ] Started LSB: Set the CPU Frequency Scaling governor to "ondemand".
[ OK ] Started LSB: centralized coordination service.
[FAILED] Failed to start Contrail controller configuration.
See 'systemctl status contrail-ansible.service' for details.
[ OK ] Reached target Multi-User System.

root@kolla2:~# docker exec -it controller bash

root@kolla2(controller):/# ss -l4
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
udp UNCONN 0 0 *:bootpc :
udp UNCONN 0 0 172.20.0.64:ntp :
udp UNCONN 0 0 127.0.0.1:ntp :
udp UNCONN 0 0 *:ntp :
tcp LISTEN 0 128 *:ssh :
tcp LISTEN 0 50 *:8888 :
tcp LISTEN 0 128 127.0.0.1:6010 :
tcp LISTEN 0 50 127.0.0.1:61619 :
tcp LISTEN 0 50 *:61620 :

root@kolla2(controller):/# systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● contrail-ansible.service loaded failed failed Contrail controller configuratio

root@kolla2(controller):/# contrail-status
== Contrail Control ==
contrail-control: inactive (disabled on boot)
contrail-named: inactive (disabled on boot)
contrail-dns: inactive (disabled on boot)
contrail-control-nodemgr: inactive (disabled on boot)
== Contrail Config ==
contrail-api: inactive (disabled on boot)
contrail-schema: inactive (disabled on boot)
contrail-svc-monitor: inactive (disabled on boot)
contrail-device-manager: inactive (disabled on boot)
contrail-config-nodemgr: inactive (disabled on boot)
== Contrail Web UI ==
contrail-webui: inactive (disabled on boot)
contrail-webui-middleware: inactive (disabled on boot)
== Contrail Support Services ==
zookeeper: inactive
rabbitmq-server: inactive (disabled on boot)

root@kolla2(controller):/# systemctl start contrail-ansible.service
Job for contrail-ansible.service failed because the control process exited with error code. See "systemctl status contrail-ansible.service" and "journalctl -xe" for details.
root@kolla2(controller):/# journalctl -xe
-- Subject: Unit contrail-ansible.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit contrail-ansible.service has begun starting up.
Mar 21 13:33:48 kolla2.mydomain.com contrailctl[1376]: PLAY [Setup Controller]********************************************************
Mar 21 13:33:48 kolla2.mydomain.com contrailctl[1376]: TASK [setup]*******************************************************************
Mar 21 13:34:19 kolla2.mydomain.com contrailctl[1376]: An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TimeoutError: Timer expired
Mar 21 13:34:19 kolla2.mydomain.com contrailctl[1376]: fatal: [localhost]: FAILED! => {"changed": false, "cmd": "/bin/lsblk --list --noheadings --paths --output NAME,UUID", "failed": true, "msg": "Timer expired", "rc": 257}
Mar 21 13:34:19 kolla2.mydomain.com contrailctl[1376]: to retry, use: --limit @/contrail-ansible-internal/playbooks/contrail_controller.retry
Mar 21 13:34:19 kolla2.mydomain.com contrailctl[1376]: PLAY RECAP *********************************************************************
Mar 21 13:34:19 kolla2.mydomain.com contrailctl[1376]: localhost : ok=0 changed=0 unreachable=0 failed=1

Mar 21 13:34:20 kolla2.mydomain.com systemd[1]: contrail-ansible.service: Main process exited, code=exited, status=1/FAILURE

Mar 21 13:34:20 kolla2.mydomain.com systemd[1]: Failed to start Contrail controller configuration.

-- Subject: Unit contrail-ansible.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit contrail-ansible.service has failed.

-- The result is failed.
Mar 21 13:34:20 kolla2.mydomain.com systemd[1]: contrail-ansible.service: Unit entered failed state.
Mar 21 13:34:20 kolla2.mydomain.com systemd[1]: contrail-ansible.service: Failed with result 'exit-code'.

root@kolla2(controller):/# /bin/lsblk --list --noheadings --paths --output NAME,UUID
lsblk: dm-0: failed to get device path
lsblk: dm-1: failed to get device path
lsblk: dm-0: failed to get device path
lsblk: dm-1: failed to get device path
/dev/sda
/dev/sda1
/dev/sda2
/dev/sda5
root@kolla2(controller):/#

root@kolla1:~# cat /etc/hosts
127.0.0.1 localhost
127.0.1.1 kolla1

The following lines are desirable for IPv6 capable hosts

::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

BEGIN ANSIBLE GENERATED HOSTS

172.20.0.35 kolla1
172.20.0.47 kolla3
172.20.0.34 kolla4

END ANSIBLE GENERATED HOSTS

127.0.0.1 localhost
172.20.0.64 kolla2
127.0.0.1 kolla1
kolla2.mydomain.com kolla2
kolla1.mydomain.com kolla1

root@kolla1:~# ansible --version
ansible 2.4.3.0

root@kolla1:~# cat OpenContrail-Kolla/contrail-ansible/playbooks/inventory/my-inventory/hosts | grep -v "#" | grep -v ";"

[contrail-controllers]
kolla2.mydomain.com

[contrail-analyticsdb]
kolla2.mydomain.com

[contrail-analytics]
kolla2.mydomain.com

[contrail-compute]
kolla2.mydomain.com

[openstack-controllers]
kolla1.mydomain.com

root@kolla1:~# cat OpenContrail-Kolla/contrail-ansible/playbooks/inventory/my-inventory/group_vars/all.yml

docker_registry: "gokulpch"
docker_registry_insecure: True
docker_install_method: package
ansible_user: root
ansible_become: true
ansible_ssh_private_key_file: ~/.ssh/id_rsa
contrail_compute_mode: container
os_release: ubuntu16.04
contrail_version: 4.0.1.0-12
cloud_orchestrator: openstack
vrouter_physical_interface: ens3
global_config: { external_rabbitmq_servers: 172.20.0.35 }
rabbitmq_config: { user: openstack, password: contrail1 }
keystone_config: {ip: 172.20.0.35, admin_password: contrail1, auth_protocol: http}

root@kolla1:~# cat OpenContrail-Kolla/kolla-ansible/ansible/inventory/multinode |grep -v '#' |head -n 18
[control]
kolla1.mydomain.com

[network]
kolla1.mydomain.com

[compute]
kolla3.mydomain.com
kolla4.mydomain.com

[monitoring]
kolla1.mydomain.com

[storage]
kolla1.mydomain.com

root@kolla1:~# cat /etc/resolv.conf
nameserver 172.20.0.2
search mydomain.com

root@kolla1:# ping kolla2
PING kolla2 (172.20.0.64) 56(84) bytes of data.
64 bytes from kolla2 (172.20.0.64): icmp_seq=1 ttl=64 time=1.04 ms
64 bytes from kolla2 (172.20.0.64): icmp_seq=2 ttl=64 time=0.531 ms
^C
--- kolla2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.531/0.790/1.049/0.259 ms
root@kolla1:# ping kolla2.mydomain.com
PING kolla2.mydomain.com (172.20.0.64) 56(84) bytes of data.
64 bytes from kolla2 (172.20.0.64): icmp_seq=1 ttl=64 time=0.720 ms
64 bytes from kolla2 (172.20.0.64): icmp_seq=2 ttl=64 time=0.553 ms
^C
--- kolla2.mydomain.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.553/0.636/0.720/0.087 ms

root@kolla2:~# cat /etc/resolv.conf
nameserver 172.20.0.2
search mydomain.com

root@kolla2:# ping kolla2
PING kolla2 (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.141 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.132 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.134 ms
^C
--- kolla2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.132/0.135/0.141/0.014 ms
root@kolla2:# ping kolla2.mydomain.com
PING kolla2.mydomain.com (172.20.0.64) 56(84) bytes of data.
64 bytes from kolla2 (172.20.0.64): icmp_seq=1 ttl=64 time=0.070 ms
64 bytes from kolla2 (172.20.0.64): icmp_seq=2 ttl=64 time=0.130 ms
^C
--- kolla2.mydomain.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.070/0.100/0.130/0.030 ms

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions