Supporting data vs. supporting analysis/viz

[Mr0grog]

This is branched off a discussion w/ @bensheldon about yak-shaving, immediate goals, what actually needs doing to craft an effective presentation and narrative here: #39

Quick recap on the near-term goals here:

• Author a narrative that illustrates the impact of downtime through some real stories and a high-level overview monitoring data.
• Author a technical write up for government and civic tech folks looking to implement monitoring on services they care about.
• Present the story at a health & technology conference in Boston on April 1.

(Longer term goals, which might benefit more from more squeaky-clean code, left out for now.)

To do all that, need to feel confident we are writing and building on reliable data. What’s the data/what do we need to support narrative and visualization?

1. Data that we feel confident in. Can we pull it reliably from backing services like Pingometer? Do we feel confident that what we record in the event hook is accurate? (e.g. doesn’t randomly have holes because the process crashed or timed out or something.) etc.
2. Snapshots of sites when down (also up). As many as possible, in all the varying ways they can be down. Seeing a sad, broken site is compelling. Seeing a wall of screenshots of sad, broken sites is especially so. (This issue has been kind of a crazy iceberg.)
3. If possible, our reliability/availability shouldn’t be hindered by that of supporting services (e.g. Pingometer) or the sites we’re monitoring. This is basically Data Caching? #17.
4. Easy ability to query the data for:
   • Screenshots
   • Times sites were down
   • Current status of sites (up/down, maybe load time?)
   • Down/uptime over various timeframes (today, past week, past month, etc) or as time series. Might not need this to be backend; could probably calculate easily enough in JS given a easy-to-analyze blob of JSON data from the server. See also Show day/week aggregations #3.
   • Basic info about the sites we are monitoring: Name, state, host/url, etc. Would be super-cool to have more meta-info about who built and who hosts the sites, when they were made, what services they support (e.g. just info? just application? balance checking?), planned downtimes, and so on. Getting the actual data is more research (looking here at @bengolder, @lippytak, @alanjosephwilliams), but making a place for it or making it possible to add easily should the above people feel it’s useful in the narrative is more what I mean.
5. API or daily data dumps or something in service of Provide public access to underlying monitoring data? #15.
6. Place to put the various write-ups and transform them into a web page. Maybe a folder full of markdown that gets read or compiled and presented on a page? Maybe its just a static page and the work happens on Google Docs or in the repo wiki or somewhere else. See Write an awesome article #6, I guess.
7. Place to write/track tech docs/description/methods stuff from Tech docs/description in the readme or wiki #12. As above, maybe not technical at all. Maybe this is just the wiki for now.

That’s what I’ve got for now. It’s a little high-level and a little stream-of-consciousness and not very thought through. Anyone should feel free add to the list, pare it down, clarify, and ask for more details.