1 ping per tick

[Dieterbe]

at the summit in March we decided that instead of our current approach:

• each tick (10/60s whatever), send 5 pings "in a row", track min/max/avg/...
  we can better just do:
• tick more frequently, and send exactly 1 ping at each tick. and just store the result of that ping.

this way we have more accurate, fine grained info, and we can still get min/max/.., in fact this makes it more elegant to compute the summary stats across arbitrary intervals.

[nopzor1200]

I don't think we can consider this until post roll-ups.

Without roll-ups, it seems it'd just make generating our current Dashboards take 10x as long

[Dieterbe]

maybe. keep in mind the amount of points due to increased granularity would be about the same as the decreased points due to no longer having min/max/avg/.. metrics. not sure how that translates into graphite performance. we would also have to rethink our ping graphs for this. and a histogram-style plot would probably make more sense.

[nopzor1200]

even on our default 3 hour dashboards, won't we still have to average datapoints together (and process every datapoint) in order to avail ourselves of maxdatapoints?

at 1 second resolution anything more than say ~16 minutes (ie. about 1000 pixels), depending on your screen rez, of raw data is going to get consolidated.

Every datapoint is going to get processed to do the consoliation, right? I guess I'm confused why we wouldn't have to average metrics.

[nopzor1200]

Also.. to clarify..

When you say histogram-style plot do you mean like a heatmap (where the x axis is still time?)

Or, do you mean histogram as in showing the distribution of values over the time range, with the x axis being values and the y axis being the frequency of how often those values occur over the time range?

?

@Dieterbe

[Dieterbe]

yes i meant histogram-over-time aka heatmap. I think that's the ultimate best viz and the sweet spot of data resolution here is roughly about 5-10x more points than you have pixels, I think. (that way you can draw buckets of 10 pix wide that each represent 50 to 100 points)
i've emailed about this a long time ago, but I think ultimately we should use heatmaps to viz higher data densities

anyway, that is unfortunately quite a ways off, and yes your numbers seem right given our current graph widgets. i never said we wouldn't have to average (or aggregate metrics), not sure where you got that from, i said this makes it more elegant to compute the summary stats across arbitrary intervals (this includes graphite aggregations).
so basically i was agreeing that this would be a bit tougher on graphite in that it now has to pull in (and process/aggregate) a more dense series, but at least it only has to pull in 1 series, instead of the current 5.

the harder problem of course is how do you roll this data up so that you can still do accurate heatmaps on aggregated data. I think OmniTI may have mentioned somewhere how they do this, i'm sure there's some papers on this topic.

anyway i was probably not clear enough in the original comment, i'm not suggesting implementing this straight away, but it is a thing that i thought had to be documented in the new repo so we don't forget about it.

[nopzor1200]

After your clarifications I guess I agree 100% with everything you're saying.