Improve handling of CP restart in receive loop

[ibaldin]

Here's the sequence where things go wrong (for me):

• register receiver
• loop forever
• recvevent

All good so far.
Control plane restarts (this was a test we ran on purpose)

No grpc status reports arrive at control plane... control plane evicts the receiver.
Receiver appears to be stuck forever in recvevent... because we're no longer sending events at it because it isn't providing status updates.

At least that's what I think is happening in that scenario.

[ibaldin]

More from stac:

Further testing on this on Friday revealed that the background thread doing grpc updates was actually still running and trying to send updates. Those updates were being refused by the control plane since that receiver had timed out and been evicted already (due to >5s gap in receiver updates). Derek has made some changes to the control plane to how this is handled:

• after 5s of missing receiver status updates, that receiver is marked as unavailable (but not deregistered/evicted)
• after 1m (I think) of missing receiver status updates, the receiver is deregistered/evicted

This should allow for normal control plane restarts to keep all receivers intact so they don't need to handle re-registration in their mainloop under most scenarios. Now that these 2 timeouts are independent, we can tune them to get the best results.
Separately from the above control plane changes, I changed my receive mainloop to add wait_ms=1000 to my recvEventBytes() call so that the receiver mainloop would periodically get to run and could notice the case where the receiver had gotten deregistered. I can see the stats indicating grpc errors during the control plane restart interval and then (after Derek's fixes above) I can see things recover again once the control plane is up.

For a truly robust receiver implementation (surviving long-ish outages where it gets deregistered), I think we'll want to provide the ability to auto-reregister if/when comms were restored after the 1min eviction timeout.

[ibaldin]

@howderek looking at this issue it isn't clear to me what needs to be changed (if any)? Based on what @solidgoldbomb reported, things recover when CP is restarted, while logging errors, which I think is what we want to have?