Clock sync detector should kill nodes

When the clock-sync detector sees that the time delta has exceeded the buffer in skuld.task, it should nuke the node and log an informative message. Probably the safest thing for now.

[danielcompton]

Do we need to reach a consensus between nodes of where the time is?

All nodes regularly exchange heartbeat messages with their current clock. If any differ by more than a few seconds, the furthest ahead kills itself.

This would mean that one node with a lagging clock could kill all the others

[danielcompton]

Probably want to check this on startup too.

[eric]

I would definitely be interested in any papers on loosely coordinating clocks. I'm sure there are some smart people who have already solved this better than I can think of.

[danielcompton]

NTP is the prior art that comes to mind first, I want to think about whether it's appropriate for our use case though.

[aphyr]

The clock skew detector is also about message latency, not just clock sync. NTP is solving a subtly different problem.

[aphyr]

I'm reasonably confident that lagging clocks should result in unavailability; if we bias towards the future, you can wind up invalidating some of the weak-time-ordered claim semantics.