Central worker recovery should not be triggered because of unacknowledged messages

[danfilip]

Prolog:

worker recovery is checked every 2 minutes by Central (WorkerRecoveryServiceImpl.java). A worker is deemed to be recovered if
(the worker has not sent a keepalive recently) OR (the queue contains unacknowledged messages for this worker)

What happens:

1. there is only one worker in the RAS_Operator_Path group alias. the worker can connect to central fine...there is no connection issue.
2. queue contains 8 older and unacknowledged messages by the above RAS
3. every 2 minutes, central triggers recovery of this worker, which leads to the triggering of the recovery/restart of the worker itself(due to WRW change)
4. the above process obviously stops all running flows on the worker, slowing all executions down on central.
5. because there is only this worker in the RAS_Operator_Path, and he was already determined as "not alive" by the central recovery mechanism, the unack. messages can't be assigned to this worker(despite him being very much alive).
   this makes it impossible to reassign those 8 messages, and the recovery for this worker will be attempted every 2 minutes, forever...until there will be another worker in the RAS_Operator_Path
6. operations that need more than 2 min, will NEVER complete, and their flows remain stuck

Suggestions:

worker recovery should not be triggered by the presence of unack queue messages. if the worker is alive(has sent keep-alives) and he's the only one in that group, it should be attempted to reassign those messages to him again, and not trigger the recovery process, which will cause all flow runs on that worker to be brutally destroyed.

[natabeck]

In case we have un acked messages and we dont have additional worker to assign the messages to - then those flows are supposed to go to PAUSED - NO_WORKER_IN_GROUP status. Once the recovery of the worker is finished the user can resume those flows. The behaviour that is described above is not expected - I think there is some issue with that specific environment.

[meshipeer]

why this is here?

[danfilip]

we could at least improve logging and debug the queue message id's in the log. this way we would know that if the same messages are printed as unassigned, that there is a corruption.

[danfilip]

hi all.

attaching simple and effective change of reading unack messages.
only changed the sql query to join the queues with the states table , so that we only read queues records that have a corresponding record in the states table.

also removed the original subselect with a more efficient group by clause, achieving the same result.

new select looks like this:

final private String QUERY_COUNT_MESSAGES_WITHOUT_ACK_FOR_WORKER_SQL =
        "SELECT COUNT(*) FROM " +
                " ( " +
                "   SELECT COUNT(*) DUMMY " +
                "   FROM  OO_EXECUTION_QUEUES  q " +
                "   INNER JOIN OO_EXECUTION_STATES st " +
                "   ON q.EXEC_STATE_ID = st.ID " +
                "   WHERE " +
                "   q.ASSIGNED_WORKER  = ? " +
                "   GROUP BY q.EXEC_STATE_ID " +
                "   HAVING MAX(q.STATUS) = ? and MAX(q.MSG_VERSION) < ? " +
                " ) SUBSELECT ";

Patch file here:
unack_read_consistent_records.patch.txt