EW-9372 EW-9455 [o11y] Report SpanOpen event earlier #5370

fhanau · 2025-10-21T20:36:46Z

Purpose:
This PR serves to perform two long-standing cleanup tasks in the STW implementation:

Sending the SpanOpen event as soon as a span is opened instead of when it closes
Getting rid of the CompleteSpan struct, which represents a full span but is something that won't be needed once SpanOpen is handled separately.

To implement this in a backwards-compatible way, we need to land it in two parts so that the old code path and the new code path are both supported until we have phased out the old version which doesn't have the APIs for handling SpanOpen separately.

For code that is workerd-only and thus never involved in RPC or that is solely on the RPC server side, we can already decompose function calls so that we don't need to implement the sape functionality twice. This needs to land alongside a downstream PR. A follow-up PR will actually invoke the code path to send SpanOpen first, get rid of CompleteSpan struct and perform a bunch of cleanup – see #6051.

Note that:

The internal tracing system will not be affected by these changes – we still propagate completed spans there. In the final version, this differentiation is implemented through differences in the SpanObserver implementations.
Some functions that are being added here won't actually be called just yet, that will change in the follow-up and in some cases they are already necessary based on backwards-compatibility.
Commit history still needs to be cleared up

codspeed-hq · 2025-10-22T21:46:42Z

Merging this PR will not alter performance

✅ 70 untouched benchmarks
⏩ 129 skipped benchmarks¹

_{Comparing felix/102125-stw-cleanup (cc51938) with main (26d1b59)}

129 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

mar-cf · 2025-10-28T08:29:21Z

Tests fail, but that just might be from being out of sync or ontop of something old?

A short PR description would help.

fhanau · 2025-12-24T17:59:00Z

Closing this for now – delivering SpanOpen earlier would make it more difficult to implement renaming spans, which we may support in the future.

danlapid · 2025-12-25T15:32:23Z

We should definitely reopen this and send SpanOpen events when the spans open and not when they close.
That is a key goal of Streaming Tail Workers compared to Buffered Tail Workers, we should not lose sight of that.
The rename does not relate to this.
OTEL officially supports a UpdateName message (https://opentelemetry.io/docs/specs/otel/trace/api/#updatename) which can be emitted at any point between the SpanOpen and the SpanClose to rename the span.
That's what we should also build into the Streaming Tail Workers protocol.

github-actions · 2026-02-10T15:08:46Z

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

codecov-commenter · 2026-02-10T15:31:46Z

Codecov Report

❌ Patch coverage is 31.13208% with 73 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.30%. Comparing base (d2c9058) to head (cc51938).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
src/workerd/io/tracer.c++	37.28%	29 Missing and 8 partials ⚠️
src/workerd/io/trace.c++	5.71%	33 Missing ⚠️
src/workerd/server/server.c++	75.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5370      +/-   ##
==========================================
- Coverage   70.35%   70.30%   -0.06%     
==========================================
  Files         408      408              
  Lines      108651   108735      +84     
  Branches    17991    18007      +16     
==========================================
+ Hits        76444    76448       +4     
- Misses      21409    21482      +73     
- Partials    10798    10805       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This PR serves to perform two long-standing cleanup tasks in the STW implementation: 1) Sending the SpanOpen event as soon as a span is opened instead of when it closes 2) Getting rid of the CompleteSpan struct, which represents a full span but is something that won't be needed once SpanOpen is handled separately. To implement this in a backwards-compatible way, we need to land it in two parts so that the old code path and the new code path are both supported until we have phased out the old version which doesn't have the APIs for handling SpanOpen separately.

fhanau · 2026-02-10T23:19:14Z

Codecov Report

❌ Patch coverage is 39.39394% with 80 lines in your changes missing coverage. Please review. ✅ Project coverage is 70.29%. Comparing base (d2c9058) to head (538c76c). ⚠️ Report is 1 commits behind head on main.

The coverage percentage appears lower than it should be here as some functions are only used upstream/not used at all until the follow-up PR. I'm convinced that our coverage doesn't actually get worse when taking that into account.

fhanau · 2026-02-10T23:36:01Z

src/workerd/io/tracer.h


  // helper method for addSpan() implementations
  void adjustSpanTime(tracing::CompleteSpan& span);
+  void adjustSpanTime(tracing::SpanEndData& span);


These will be deduplicated in the follow-up PR

fhanau · 2026-02-10T23:36:36Z

src/workerd/io/trace.h

  //
-  // This should always be called exactly once per observer.
+  // This should always be called exactly once per observer at span completion time.
  virtual void report(const Span& span) = 0;


For user tracing, report() will be used to only transmit the span end data in the follow-up. For internal tracing, this will continue to be used to transmit the full span. reportStart() is not yet used and will only used in the user tracing framework (no-op otherwise).

fhanau · 2026-02-10T23:37:40Z

src/workerd/io/trace.h

+  kj::HashMap<kj::ConstString, tracing::Attribute::Value> tags;
+
+  // Convert CompleteSpan to SpanEndData
+  explicit SpanEndData(CompleteSpan&& span);


This will also no longer be needed with the follow-up.

fhanau · 2026-02-10T23:37:59Z

src/workerd/io/tracer.c++

  }
 }

+void BaseTracer::adjustSpanTime(tracing::SpanEndData& span) {


This is copied from BaseTracer::adjustSpanTime(tracing::CompleteSpan&), will be deduplicated in the follow-up PR. Note that the operationName variable is no longer available, but if we need to debug an error here we can still infer the operation using the spanId, which maps to the span's SpanOpen event which has the operationName.

fhanau · 2026-02-10T23:38:25Z

src/workerd/io/trace.capnp

+  # Representation of an event that indicates completion of a user span. This information is
+  # provided to the streaming tail worker in the Attributes and SpanClose events.
+
+  # TODO(cleanup): startTimeNs is merely used as a fallback timestamp, consider obsoleting it.


This is only used as a fallback timestamp in the case of errors, but @mar-cf urged me to still support it.

fhanau · 2026-02-10T23:39:07Z

src/workerd/io/trace.c++

+  }
+}
+
+void SpanEndData::copyTo(rpc::SpanEndData::Builder builder) const {


You'll notice that this is also duplicated with CompleteSpan, but that will go away in the follow-up as discussed.

fhanau requested review from a team as code owners October 21, 2025 20:36

fhanau added the observability label Oct 22, 2025

fhanau force-pushed the felix/102125-stw-cleanup branch 2 times, most recently from 20b779f to b6452a3 Compare October 22, 2025 20:01

fhanau requested a review from a team as a code owner October 22, 2025 20:01

fhanau closed this Dec 24, 2025

fhanau reopened this Dec 26, 2025

fhanau marked this pull request as draft December 26, 2025 15:24

fhanau force-pushed the felix/102125-stw-cleanup branch 3 times, most recently from 7db22e8 to c73e84d Compare February 9, 2026 19:19

fhanau changed the title ~~[o11y] Report SpanOpen event earlier~~ EW-9372 EW-9455 [o11y] Report SpanOpen event earlier Feb 10, 2026

fhanau force-pushed the felix/102125-stw-cleanup branch from 6d91a06 to 538c76c Compare February 10, 2026 15:07

fhanau added 2 commits February 10, 2026 18:10

downstream-compatible version

693b134

fhanau force-pushed the felix/102125-stw-cleanup branch from 538c76c to c5c817f Compare February 10, 2026 23:14

Prepare getting rid of CompleteSpan

cc51938

fhanau force-pushed the felix/102125-stw-cleanup branch from c5c817f to cc51938 Compare February 10, 2026 23:35

fhanau commented Feb 10, 2026

View reviewed changes

fhanau mentioned this pull request Feb 10, 2026

EW-9372 EW-9455 [o11y] Report SpanOpen as soon as available, remove CompleteSpan #6051

Open

fhanau marked this pull request as ready for review February 10, 2026 23:59

fhanau requested a review from mar-cf February 10, 2026 23:59

EW-9372 EW-9455 [o11y] Report SpanOpen event earlier #5370

Are you sure you want to change the base?

EW-9372 EW-9455 [o11y] Report SpanOpen event earlier #5370

Conversation

fhanau commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

mar-cf commented Oct 28, 2025

Uh oh!

fhanau commented Dec 24, 2025

Uh oh!

danlapid commented Dec 25, 2025

Uh oh!

github-actions bot commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

fhanau commented Feb 10, 2026

Codecov Report

Uh oh!

fhanau Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

fhanau Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fhanau Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

fhanau Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

fhanau Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

fhanau Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fhanau commented Oct 21, 2025 •

edited

Loading

codspeed-hq bot commented Oct 22, 2025 •

edited

Loading

github-actions bot commented Feb 10, 2026 •

edited

Loading

codecov-commenter commented Feb 10, 2026 •

edited

Loading

fhanau Feb 10, 2026 •

edited

Loading