Apply a timeout on metrics collection

[arnedc]

Most Prometheus clients apply a timeout on metrics collection, but this doesn't seem to be the case for the Ocaml prometheus client.
The Prometheus scrape requests even add a specific header (X-Prometheus-Scrape-Timeout-Seconds) to each scrape request defining the timeout that is used at the server-side.

However, I assume that the timeout only closes the connection when no response is received within this timeout and I guess the Prometheus client already sends back a response even when all metrics are not collected yet. Because I'm seeing scrape durations (up to 1000s) exceeding largely the scrape timeout (30s).

I'm guessing it would be a good idea to apply a timeout on metrics collection and perhaps also try to only send a response when metrics collection is finished.

[talex5]

Are you using some custom collectors? The default ones aren't doing anything that takes any significant time (versions before 0.7 did some GC stats that could be expensive on large heaps, but that's fixed now). All it's doing it formatting some numbers as a string.

Perhaps your program is blocked for some other reason, and therefore unable to respond to the collection request?

[arnedc]

I probably should have mentioned that I'm using version 0.4.2, so getting the GC stats might play up. I'm planning on updating the version though.
Also, the program is heavily loaded so it might indeed be that it's not able to respond in time.
Actually, that would even be another reason for putting a timeout on the collection, as we don't want it to steal valuable resources from a busy process.