Add comprehensive tracing

[guseggert]

Master issue for adding comprehensive tracing throughout the go-ipfs codebase.

Some requirements:

• End users and gateway operators should be able to trace costly work units throughout the various parts of the go-ipfs internals
• Should fit into a larger distributed tracing story if go-ipfs is part of a larger system
• Traces all the types of subsystems
  • Synchronous work done as part of requests such as gateway requests
  • Async workers that service sync requests
  • Async workers that aren't involved in synchronous requests
    • DHT background work
    • Work done at daemon startup
  • libp2p
• Pluggable into different tracing backends
• Easy to instrument existing un-instrumented code
• Easy to instrument new code such as plugins, new interface implementations, etc.
• Random sampling knob to tune perf/observability tradeoff

I plan to implement this against OpenTelemetry. Tactically I want to start at a high level (e.g. gateway reqs) and iteratively work down into the guts, driven by trying to answer questions about real requests so that we are instrumenting the right things.

Some related issues:

• context/tracing in the blockstore/datastore pipeline #6803
• Towards better tracking of gateway performance #5783

Todos:

• Add tracer provider in go-ipfs w/ exporter configs, and add gateway and CoreAPI tracing
  • feat: add basic gateway tracing #8595)
• Deploy to public gateways
• Add tracing in subsystems (incomplete list)
  • go-bitswap
  • go-namesys
  • go-datastore
  • go-libp2p-kad-dht
  • libp2p subsystems
• The Provider API needs a context plumbed through to go-ipfs-provider

[BigLep]

@guseggert: a couple of quick questions on this:

1. Do we have a sense of how much effort from you this will take in terms of ideal dev days/weeks? (I'm asking so can figure out when this may be of benefit for Bifrost)
2. After this work is done, what additional work will Bifrost need to do before they're able to see tracing data of a specific request?

[guseggert]

1. We can start getting local barebones traces back with 3 days of dev effort, though they won't go deep into the stack (just the glue in go-ipfs). Also not including the initial collector setup below, which is required for tracing on deployed infra. Adding additional spans deeper in the stack is trivial, with some minor overhead for plumbing that back into go-ipfs...in aggregate it'd likely take a couple weeks to add basic instrumentation to major components like Bitswap, DHT, datastores, other libp2p pieces, etc., and get them all plumbed up into go-ipfs.

2. I met with @coryschwartz and @frrist last week, we discussed 2) (amongst other things). I'd estimate the effort to integrate this with our infra at 1 week. The consensus was to use Grafana Tempo, which is a hosted OpenTelemetry collector, which Grafana has a UI on top of (see https://grafana.com/docs/tempo/latest/getting-started/tempo-in-grafana/). We'd follow the onboarding instructions here: https://grafana.com/orgs/protocollabs/hosted-traces/76.

[BigLep]

@guseggert : thanks! With local barebones traces, what kind of metrics will we get?

[guseggert]

We can get timing info and some basic request metadata for BlockStore, BlockService, DAGService, PeerStore, Pinner, and IPNS, possibly others.

Simple example, here's a req I recorded locally showing what's happening as go-ipfs builds an index page for dist.ipfs.io, from a few spans in the gateway request handler and core

[proto-sid]

cc: @gmasgras for visibility

[guseggert]

Draft PR for gateway tracing: #8595

[guseggert]

Netops has a summit this week they are busy with, so things will be slow this week on infra side. @thattommyhall is working on code changes to install a collector on the public gateway hosts.

[proto-sid]

@thattommyhall Could you share the current state of this implementation?