Skip to content

Services are not running when bringing up node after shutdown in B-series cluster #169

@pradvara

Description

@pradvara

Steps to reproduce:

Add nodes to cluster

[stack@contiv-b1 ~]$ clusterctl nodes get | grep " status"
contiv-b1-FCH1702J22M: status: Allocated
contiv-b2-FCH1701J2KV: status: Allocated
contiv-b3-FCH1828KBGQ: status: Allocated
contiv-b4-FCH1811JLXV: status: Allocated
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b6-FCH1811JD9C: status: Allocated

Shutdown a node:

[stack@contiv-b1 ~]$ clusterctl node get contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: Inventory State
contiv-b5-FCH1834JF2M: name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: prev_state: Discovered
contiv-b5-FCH1834JF2M: prev_status: Allocated
contiv-b5-FCH1834JF2M: state: Disappeared
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b5-FCH1834JF2M: Monitoring State
contiv-b5-FCH1834JF2M: label: contiv-b5
contiv-b5-FCH1834JF2M: management_address: 10.106.240.109
contiv-b5-FCH1834JF2M: serial_number: FCH1834JF2M
contiv-b5-FCH1834JF2M: Configuration State
contiv-b5-FCH1834JF2M: host_group: service-worker
contiv-b5-FCH1834JF2M: inventory_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: inventory_vars:
contiv-b5-FCH1834JF2M: node_addr: 10.106.240.109
contiv-b5-FCH1834JF2M: node_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: ssh_address: 10.106.240.109

The state of the node moves to "Disappeared"

Bring up the node again

[stack@contiv-b1 ~]$ clusterctl node get contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: Inventory State
contiv-b5-FCH1834JF2M: name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: prev_state: Disappeared
contiv-b5-FCH1834JF2M: prev_status: Allocated
contiv-b5-FCH1834JF2M: state: Discovered
contiv-b5-FCH1834JF2M: status: Allocated
contiv-b5-FCH1834JF2M: Monitoring State
contiv-b5-FCH1834JF2M: label: contiv-b5
contiv-b5-FCH1834JF2M: management_address: 10.106.240.109
contiv-b5-FCH1834JF2M: serial_number: FCH1834JF2M
contiv-b5-FCH1834JF2M: Configuration State
contiv-b5-FCH1834JF2M: host_group: service-worker
contiv-b5-FCH1834JF2M: inventory_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: inventory_vars:
contiv-b5-FCH1834JF2M: node_addr: 10.106.240.109
contiv-b5-FCH1834JF2M: node_name: contiv-b5-FCH1834JF2M
contiv-b5-FCH1834JF2M: ssh_address: 10.106.240.109

The state is Discovered, and status is allocated.

Check wheather services are running in node

[stack@contiv-b5 ~]$ ps -ef | grep netplugin
stack 2992 2785 0 15:50 pts/0 00:00:00 grep --color=auto netplugin

[stack@contiv-b5 ~]$ systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/docker.service.d
└─env.conf
Active: inactive (dead)
Docs: https://docs.docker.com

[stack@contiv-b5 ~]$ sudo systemctl status etcd
● etcd.service - Etcd
Loaded: loaded (/etc/systemd/system/etcd.service; static; vendor preset: disabled)
Active: inactive (dead)

Services are not running, but the status is allocated

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions