Proper way to perform head sampling?

[ArniDagur]

How should I perform head sampling to ensure that only a subset of traces get sampled?

With tracing + tracing-opentelemetry, you can use Sampler::ParentBased combined with Sampler::TraceIdRatioBased to sample a percentage of root spans without any parent, or 100% of child spans if the parent is sampled.

Fastrace seems to set sampled: true on every new span created with SpanContext::random.

I was considering having a new_root function in our code which would be a wrapper around Span::root which changes .sampled depending on some criteria (e.g. coin toss). However, that wouldn't work since we don't have control over all usages of Span::root. Some are in fastrace-tonic for example.

A solution to this is a blocker for us to adopt fastrace, since on high QPS services we could only capture a miniscule percentage of traces.

[andylokandy]

Seems that a proper solution is adding a sampling configuration to fastrace-tonic to randomly set sampled to false regardless of traceparent. Another way is to use tail-sampling which is highly recommended especially for high QPS scenario because we've found that the information provided by tail sampling (based on time or other criteria) is way more useful than random head sampling, especially given that tail-sampling has very little performance penalty in fastrace. You may try to use tail sampling first, and turn to the first solution if the performance is not satisfied.

[ArniDagur]

Would tail sampling be done with a custom Reporter?

I'd imagine it'd work like this: Given a list of spans, group by trace ID, find the root spans, and make a sample/no sample decision based on e.g. latency histograms and presence of failures in child spans.

I see a few problems though:

• Since the signature takes a Vec<SpanRecord>, I'm worried about what happens if a single trace is split into multiple calls to report. Then the later calls to report will have spans with a root-less trace ID. I guess you'd have to make your Reporter stateful (remember which trace IDs you decided to sample).
• There's no way to tell if a root span from e.g. fastrace-tonic is a "real" root span where no traceparent header was given, or if it's simply the "process local" root span, where the real root exists in the calling service. In the former case, the sampling decision should happen in our process, but in the latter case the sampling decision has already been made and we need to follow its lead—ensure tail sampling doesn't exclude any spans which the caller expects us to sample (or include any spans where we don't have caller information, if you don't want that).

Another way is to use tail-sampling which is highly recommended especially for high QPS scenario

we've found that the information provided by tail sampling (based on time or other criteria) is way more useful than random head sampling

Is "we" in this sentence a company? Was the tail sampling implementation you used open source or internal? If you have a working solution I'd love to see it :)

---

I know OpenTelemetry Collector also has a tail sampling processor, but I'm worried the QPS would be too high if we sent every span to OpenTelemetry Collector.

[ArniDagur]

Seems that a proper solution is adding a sampling configuration to fastrace-tonic to randomly set sampled to false regardless of traceparent

I don't think this will scale. If fastrace-tonic is your only external dependency, then perhaps. But if fastrace gains adoption you'll never be able to control all the places your library dependencies create root spans.

In addition, some organizations may want to use custom head sampling strategies (e.g. adaptive sampling), while others want a simple ratio of all roots. I don't think baking sampling strategies into fastrace-tonic or any other specific library is a good idea.

[andylokandy]

Tail sampling is a builtin feature of fastrace, which can be enabled in Config. With tail sample enabled, you can call Span::cancel to dismiss the entire trace based on your strategy.

[andylokandy]

Is "we" in this sentence a company? Was the tail sampling implementation you used open source or internal? If you have a working solution I'd love to see it :)

Fastrace is initially developed for extensive traces in DBMS, which has crazy QPS and latency requirement. Anyway, fastrace has been adopted by various projects, which can be found here, and from the results and observation, we has such conclusion.

[ArniDagur]

Tail sampling is a builtin feature of fastrace, which can be enabled in Config. With tail sample enabled, you can call Span::cancel to dismiss the entire tracing based on your strategy.

Manually calling Span::cancel from within application code is too invasive, and would only allow simple tail sampling scenarios (e.g. based on duration of root span). I want developers to be able to just run an init function in main, slap some #[trace] attributes onto the functions they want traced, set up span roots where needed, and have everything work. That's how it is with tracing + tracing-opentelemetry.

Also, how is the second issue I raised tackled? A sampling decision from one service has to be able to propagate to all downstream services. I said:

There's no way to tell if a root span from e.g. fastrace-tonic is a "real" root span where no traceparent header was given, or if it's simply the "process local" root span, where the real root exists in the calling service.

Thank you for the help :)

[andylokandy]

I want developers to be able to just run an init function in main, slap some #[trace] attributes onto the functions they want traced, set up span roots where needed, and have everything work. That's how it is with tracing + tracing-opentelemetry.

In fact, tracing falls short here. The random root is easy to get started but in real production will leave you massive disconnected and useless traces/spans. You can see detailed examples here. In the real production that tracing be really useful (only from several examples that I've witness or participated in, including positive examples and negative ones), propagation/sampling/root has to be carefully designed and maintained. If you'd like to slap some attribute on random functions and imagine it'll provide something useful when the system becomes abnormal online, you may be disappointed.

There's no way to tell if a root span from e.g. fastrace-tonic is a "real" root span where no traceparent header was given, or if it's simply the "process local" root span, where the real root exists in the calling service.

Reasonable. I'd like to add a config to fastrace-tonic to allow user to control the root span construction behavior.

[ArniDagur]

The random root is easy to get started but in real production will leave you massive disconnected and useless traces/spans.

propagation/sampling/root has to be carefully designed and maintained. If you'd like to slap some attribute on random functions and imagine it'll provide something useful when the system becomes abnormal online, you may be disappointed.

There's nothing preventing you from both carefully designing and maintaining trace propagation, while still using implicit root span creation. In fact, with implicit roots you have better visibility into where your actual trace roots are (they're obvious from seeing disconnected spans), giving you the opportunity to improve on trace propagation. When manually specifying root spans, you have to guess all the right places to instrument, and you'll almost certainly be missing some. My previous company used this approach to trace requests across dozens of micro-services.

However, that was not my actual point. In fact, root spans being explicit is in some way nice, as it allows you to grep your codebase for all the roots. My point was that tail sampling by manually calling Span::cancel is not scalable in terms of developer time and effort. Likewise, I don't see how it solves most use cases other than just canceling based on duration.

I'd like to add a config to fastrace-tonic to allow user to control the root span construction behavior.

What kind of API are you considering?

This sounds like a step in the right direction. However, it doesn't address the entire issue. If fastrace becomes popular, and downstream libraries start using it, there's no way to ever control all the places Span::root is called. It'd only take one bad dependency capturing 100% of traces to completely DDoS your OpenTelemetry Collectors with traffic.

What would solve the issue is a global mechanism similar to opentelemetry::sdk::trace::Sampler and opentelemetry::sdk::trace::ShouldSample to control head sampling, even when traces are initiated in libraries you don't control.

[andylokandy]

There's nothing preventing you from both carefully designing and maintaining trace propagation, while still using implicit root span creation. In fact, with implicit roots you have better visibility into where your actual trace roots are (they're obvious from seeing disconnected spans), giving you the opportunity to improve on trace propagation. When manually specifying root spans, you have to guess all the right places to instrument, and you'll almost certainly be missing some. My previous company used this approach to trace requests across dozens of micro-services.

There are two feature requests:

1. Auto start root, or some methods to detect spans without proper root.

There is workaround on current api: detect whether current parent exist and start span with the current span or create a root. But still it's not recommended. Because, by design, library could annotate #[fastrace::trace] everywhere and let the application/downstream to decide whether to enable the spans by wrapping the calls to library with Span::root. If all spans unconditionally start roots, the application/downstream has no change to ever shut the spans down.

2. An global level override for the sampled flag on SpanContext for Span::root.

It's possible to implement such a mechanism in fastrace. But would it better if you can calculate the head sampling flag in the client side (i.e., the sampled in traceparent)?