Distinguish between monitors with "no data" and those that are "not down" for a given timeframe

[alanjosephwilliams]

Our monitor has been showing that Vermont has been down since 03/24/15 at 6:15 AM. It's a basic HTTPS monitor with no transaction.

For the past two days, we've manually verified that the same URL (with the port removed) is available and loading without significant latency. (https://mybenefits.ahs.state.vt.us/Login.aspx).

@pingometer could you help us investigate?

[pingometer]

Sure, absolutely!

It looks like the monitor is checking for "User ID", but the page has been changed to say "Email".

If you change the Expect Data to "Email" or something else currently on the page, it should come back up.

[alanjosephwilliams]

Thanks @pingometer! We'll note in our write-up that we have dependencies on persistent strings on both HTTPS and transactional checks.

[daguar]

@alanjosephwilliams 👍 to noting that. What are we reporting for the downtime for VT? This strikes me as a false positive and that we might want to say it was not down.

[alanjosephwilliams]

Nope, updating the string resolved the issue:

[daguar]

Sorry to clarify what I'm saying is that it appears that we're reporting that it's been down on the last-week chart and we should consider manually overriding that.

[alanjosephwilliams]

We will not report publicly that VT was down without communicating our best understanding of the results. I'm open to "manual override" to present it as up for the duration in question as a solution, but I'm also open to something approximating "no data" between the last successful check and our resolution of the issue.

[alanjosephwilliams]

@Mr0grog any thoughts on relative LOE between those approaches, or others you might have in mind?

[Mr0grog]

Ah, this brings up the subtle distinction between “no data for timeframe X” and “not counting as down (i.e. up) for timeframe X” that I hadn’t been thinking about in the discussion on #46. Presenting bad timeframes as “up” is definitely much easier than presenting them as “no data.”

In most situations, I think distinguishing between “up” and “unknown/uncounted time” should be do-able, but it adds some overhead (in both design and implementation) to every calculation and chart we do: how does it count in the math? How is it displayed if we are displaying individual events or downed timeframes? Etc.

[alanjosephwilliams]

@Mr0grog I'm going to update this issue to reflect the current concern articulated in your last comment—our need to draw a distinction between "no data" and "not down". We'll want to scope out implementation of this along with #55 and #56 soon as we regroup and set an agenda for the project moving forward.

[Mr0grog]

FWIW, we do distinguish in the maps and “api” right now. That is:

• If we have no data for a state during the entire time period being requested, we return null, which, in terms of the map, simply doesn’t fill in the state at all (i.e. it’s white).
• If we have data for only part of the requested time period, we return an uptime calculation as a percentage of the time that we have valid data for. i.e. if we asked for one day and we had invalid data for half the day, we’d return uptime as a percentage of 12 hours instead of 24 hours. (This is similar to Pingometer’s approach, although they do it off every check and we do it off events.)

There’s still room for more nuance there—in the second case, is that right? Should we just return null if any part of the timeframe is invalid? I think that might be more a question for #55, though. For the most part, we are distinguishing and I think that probably addresses this issue.

[alanjosephwilliams]

Got it. Thanks for laying this out.

Another approach would be to visually flag a state when its respective
uptime is calculated from available data that is less than the full period
requested. An asterisk, whether literally or figuratively.

On Tue, Apr 7, 2015 at 9:24 PM, Rob Brackett notifications@github.com
wrote:

FWIW, we do distinguish in the maps and “api” right now. That is:

• If we have no data for a state during the entire time period being
  requested, we return null, which, in terms of the map, simply doesn’t
  fill in the state at all (i.e. it’s white).
• If we have data for only part of the requested time period, we
  return an uptime calculation as a percentage of the time that we have
  valid data for. i.e. if we asked for one day and we had invalid data
  for half the day, we’d return uptime as a percentage of 12 hours instead of
  24 hours. (This is similar to Pingometer’s approach, although they do it
  off every check and we do it off events.)

There’s still room for more nuance there—in the second case, is that
right? Should we just return null if any part of the timeframe is
invalid? I think that might be more a question for #55
#55, though. For the
most part, we are distinguishing and I think that probably addresses this
issue.

—
Reply to this email directly or view it on GitHub
#52 (comment)
.

t: @alanjosephwilli
p: 817 713 6264