Histogram sum is 0 most of the time.

[dmy-gh]

I there.
Trying to figure out if this library could replace telemetry_metrics_prometheus due its metrics generation resource consumption I'm facing a non expected behavior.

The following configuration:

distribution(
  [:repo, :query, :decode_time, :seconds],
  description: "The time spent decoding the data received from the database.",
  event_name: [:whatever, :repo, :query],
  measurement: :decode_time,
  reporter_options: [peep_bucket_calculator: Whatever.Bucket],
  unit: {:native, :second}
)

defmodule Whatever.Bucket do
  use Peep.Buckets.Custom,
    buckets: [0.000001, 0.000005, 0.00001, 0.00005, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 2, 3, 4, 5]
end

Produces the following Prometheus data:

# HELP repo_query_decode_time_seconds The time spent decoding the data received from the database.
# TYPE repo_query_decode_time_seconds histogram
repo_query_decode_time_seconds_bucket{le="1.0e-6"} 2957
repo_query_decode_time_seconds_bucket{le="5.0e-6"} 10746
repo_query_decode_time_seconds_bucket{le="1.0e-5"} 11002
repo_query_decode_time_seconds_bucket{le="5.0e-5"} 11067
repo_query_decode_time_seconds_bucket{le="0.0001"} 11068
repo_query_decode_time_seconds_bucket{le="0.0005"} 11069
repo_query_decode_time_seconds_bucket{le="0.001"} 11069
repo_query_decode_time_seconds_bucket{le="0.005"} 11069
repo_query_decode_time_seconds_bucket{le="0.01"} 11069
repo_query_decode_time_seconds_bucket{le="0.05"} 11069
repo_query_decode_time_seconds_bucket{le="0.1"} 11069
repo_query_decode_time_seconds_bucket{le="0.5"} 11069
repo_query_decode_time_seconds_bucket{le="1.0"} 11069
repo_query_decode_time_seconds_bucket{le="2.0"} 11069
repo_query_decode_time_seconds_bucket{le="3.0"} 11069
repo_query_decode_time_seconds_bucket{le="4.0"} 11069
repo_query_decode_time_seconds_bucket{le="5.0"} 11069
repo_query_decode_time_seconds_bucket{le="+Inf"} 11069
repo_query_decode_time_seconds_sum 0
repo_query_decode_time_seconds_count 11069

Any reason or idea for the sum being 0?
Am I missing something while configuring Peep?

[rkallos]

Hello! Thank you for your interest in Peep!

I think this might be related to units. If I sum the number of samples in each bucket, using the upper bound for each bucket, I get a sum of 0.04831200000000001 seconds. Here is the code I used:

[
  {"1.0e-6", 2957 - 0},
  {"5.0e-6", 10746 - 2957},
  {"1.0e-5", 11002 - 10746},
  {"5.0e-5", 11067 - 11002},
  {"0.0001", 11068 - 11067},
  {"0.0005", 11069 - 11068}
]
|> Enum.reduce(0, fn {str, n}, acc -> acc + (String.to_float(str) * n) end)

The reason you're seeing 0 as the sum is that Peep calls Integer.to_string/1 on your value of 0.04831200000000001.

You may want to use {:native, :millisecond} or {:native, :microsecond} in your distribution configuration, which should then report sums of 48ms or 48,312μs, respectively. If you do, be sure to adjust your bucket boundaries accordingly to reflect the change in units.

[dmy-gh]

Thank you for your reply.
By convention Prometheus expects seconds as base unit so all our metrics are converted into seconds.
It also makes it more consistent at the bucket level definition as they all express bucketed observation counts of the same unit and at the graph level as there is no need to figure out in which unit a metric is exposed.
Any reason for not converting into string without losing precision?

[rkallos]

I did some reading on Prometheus's idea and promotion of adopting base units. I don't really agree with it, but that's neither here nor there.

I looked into implementing what you've asked, and there are some barriers to Peep accurately reporting sums as floating point numbers. The main reason being that Erlang's :atomics module, which Peep uses to efficiently store and manipulate distribution-type metrics, only supports integers. While this is fine for the individual buckets, this poses a problem for the sum, which is also stored using :atomics; Peep has to round sample values before adding them to the stored sum value.

Changing this is not very straightforward, either. Calculating the sum by iterating over buckets and multiplying the number of samples in a bucket with the bucket's upper bound value produces wildly inaccurate sums, with large errors accumulating from the larger buckets.

One option might be to store milliseconds or microseconds in Peep, and then to convert to base units when exporting Prometheus data. It is mildly annoying, as it will involve another conversion step besides what is done via TelemetryMetrics, but I don't really see a way for Peep to properly store very small floating point numbers with its current storage model.

I'll continue to think about your use-case. This is not as simple than I originally thought.