processLBSMessages should not return immediately if grabbing the mutex fails for one message

[a3y3]

Problem

processLBSMessages is called for a list of messages that were fetched from the LBS (or via XPENDING in recoverUnackedMessages()). Currently, from the list of messages, if a single lock fails to be acquired, the function returns before processing the other messages. The most common event in which the lock can fail to be acquired is if some other pod has acquired the lock for the message.

However, the other messages should be given an opportunity to be processed. Consider the following sequence of events:
t=0 pod p calls recoverUnackedMessages() and sees 2 messages that are un-acked.
t=1 pod q does the same.
t=2 pod p starts processing message 1
t=2 pod q tries to acquire lock for message 1, finds out that it can't, and returns without trying to process message 2
t=3 pod p crashes

At this point, pod q would have been able to grab message 2, but now it never will.

As I write this, I realize that it's a bit of an unlikely scenario, but I don't see the problem with trying to process all messages, if grabbing the lock fails for only one message.

[bburli-craft]

• This is technically impossible in the init scenario. LBS is a redis stream configured with consumers in a load balanced way. Redis stream guarantees that no two consumers will get the same message.
• In recovery, however, this is possible and I will address this as part of recoverUnackedMessages can start heartbeats for acked messages #55 since we're changing the way recovery works there.

[bburli-craft]

@a3y3 Can we close this or do you still think this is needed?

[bburli-craft]

Converted this to discussion as there is no actionable item yet.

[soham-suki]

Now that we are using XREADGROUP in recovery, I don't think the scenario I described is possible. We can close this discussion.