vm_manager response in degraded cluster

[eroussy]

Edit after initial posting:
The problem was not correctly identified. The issue now refers to

• Add a proper timeout on ceph commands (currently the command never ends if ceph doesn't respond)
• Add a proper error message when the qorum is not formed
• Make vm_manager create command work even when one node is in standby or maintenance mode (the rest of the commands works from what I see)

Original issue:

Describe the bug
When the cluster is in a degraded state (one of the hypervisors is powered off), no vm_manager command responds (start, stop, list ...)

To Reproduce

• Create a SEAPATH cluster (no matter the distribution) and configure it
• Deploy a VM in it
• Shut down one hypervisor
• Launch a vm-mgr list command on one of the other hypervisors
• The command never responds

Expected behavior
When the quorum is formed (at least two machines are up and connected), vm_manager should be able to respond correctly, even to create and deploy VMs

First investigations
It seems that vm_manager was developed with only a fully running cluster in mind. There is no mention of the word "quorum" in the code.
vm_manager should be able to detect if the quorum is formed on the current machine

• If it is: schedule the command correctly
• If it is not: fail with a meaningful message

IMO, we should even be able to revert this PR seapath/ansible#870, as vm_manager should be able to deploy a VM even when one physical machine is out of the quorum.

[dupremathieu]

What do you mean by "degraded cluster"? It works if one the three machines is down. If two machines are down we are in the case you have described.

[eroussy]

That's not what I observe in my cluster. Even if one machine is down, vm-manager doesn't respond.
This seems to be the same problem as seapath/ansible#870

I'll check again when I have some time

[eroussy]

Indeed, it may come from the fact that osd_default_min_size was not set in my cluster and fallback to 2
https://docs.ceph.com/en/latest/rados/configuration/pool-pg-config-ref/#confval-osd_pool_default_min_size

I'll check again when I have some time

[eroussy]

In fact you are write, vm-manager can work correctly even if one node is down.
There was a ceph problem on my side that got vm-manager to block

I think we should then do three things in vm_manager

• Add a proper timeout on ceph commands (currently the command never ends if ceph doesn't respond)
• Add a proper error message when the qorum is not formed
• Make vm_manager create command work even when one node is in standby or maintenance mode (the rest of the commands works from what I see)

I updated the issue message