Cyclops operator creates orphan nodes when it is not able to completely drain the node

[skaushal-splunk]

Describe the bug
A node which is not attached to any autoscaling group is created when the draining of node fails for some reason.

To Reproduce

1. Create a node which has pods in the pending state. This is one of the reasons why node draining fails. There are are other reasons why draining fails. We want to create a scenario where draining fails.
2. Create a CycleNodeRequest to create a new node and detaches the old node.
3. Create a CycleNodeStatus to drain the node.
4. Run both CycleNodeRequest and CycleNodeStatus.

Current behavior
CycleNodeRequest works first, detaches the old node and creates a new node. CycleNodeStatus is not able to drain the node because it sees some pods in the pending status.

As a result we have an old node which is not attached to autoscaling group but still has pods running on it. New node comes up but only has daemonsets running on it.

Expected behavior

1. The old node should be drained and the new node should have all the pods from the old node
2. There should not be any node which is not attached to AutoScaling group.

Kubernetes Cluster Version
v1.25

Cyclops Version
v1.7.0

[awprice]

@skaushal-splunk Are you able to clarify what you mean by "Create a node which has pods in the pending state"?

As far as I'm aware, this is unlikely to be possible - a pod can't be scheduled onto a node and still be pending? Is there another factor causing the pods to be in this state that you aren't mentioning?

[skaushal-splunk]

I meant a pod which is not in Ready state

It could be a pod which is one of these states.

1. Completed
2. Error
3. CrashLoopBackOff
4. Running but not in Ready state. ie. the readiness probe fails

In all of these scenarios, cyclops operator fails to drain the node but removes the ec2 instance from autoscaling group.

In this state, the kubernetes node has pods running but the corresponding ec2 instance is not a part of any autoscaling group.

After this state the entire node rolling stops. We have to manually drain that node and then cyclops operator continues.