Pingometer shows downtime blips every night

[lippytak]

Synchronized downtime on two separate systems suggests Pingometer artifacts. Ideas @pingometer?

[pingometer]

@lippytak It looks like there were no events (confirmed downtime) that were synchronized for the monitors during this period. The only recent down event was on 5/23 for LEADER. This is associated with checks that fail, but not enough to meet the threshold to trigger an actual event. Both monitors use the same locations, hence a similar pattern on the chart.

We examined traceroutes from the check nodes to both monitors; there is a similar series of hops through NTT (*.us.bb.gin.ntt.net). If any of these hops have an issue (dropped packets, etc.), it impacts the checks for both monitors. But, with the sensitivity setting, we require *multiple confirmed failures before triggering an alert so this is usually not a concern.

To fix this, we could implement a "double-check". If a monitor is up and a check fails (or vice-versa), it is immediately rechecked from another location before being recorded as failed. This means checks will have the same level of fidelity as events (which always require confirmation). The downside is uptime could be artificially high - minor blips that don't persist long enough to be confirmed by the re-check would never get factored into the uptime calculation. Thoughts?