[WP] Implementation of Otel 4947 thread context#48446
Open
Conversation
Gui774ume
added
changelog/no-changelog
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5891c761c2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Comment on lines
+81
to
+82
| u64 thread_offset = get_task_struct_thread_offset(); | ||
| u64 fsbase_offset = get_thread_struct_fsbase_offset(); |
There was a problem hiding this comment.
These offsets are consumed unconditionally, but the commit only requests them on amd64 (probe_ebpf.go) and the new fallback getters return ErrorSentinel (constantfetch/fallback.go), which is later materialized as 0 in constant editors. If a process sends REGISTER_OTEL_TLS_OP on an unsupported/missing-offset environment (e.g., arm64 or x86_64 kernels where BTF lookup fails), this path reads task + 0 instead of task->thread.fsbase, so OTel context extraction becomes incorrect and can mis-associate or drop span correlation. The OTel path should be disabled unless both offsets are successfully resolved.
Useful? React with 👍 / 👎.
Contributor
Files inventory check summaryFile checks results against ancestor a01a5c3e: Results for datadog-agent_7.79.0~devel.git.200.5891c76.pipeline.104610626-1_amd64.deb:Detected file changes:
|
Contributor
Static quality checks❌ Please find below the results from static quality gates Error
Gate failure full details
Static quality gates prevent the PR to merge! Successful checksInfo
16 successful checks with minimal change (< 2 KiB)
On-wire sizes (compressed)
|
Regression DetectorRegression Detector ResultsMetrics dashboard Baseline: a01a5c3 Optimization Goals: ✅ No significant changes detected
|
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +3.61 | [+0.54, +6.67] | 1 | Logs |
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | docker_containers_cpu | % cpu utilization | +3.61 | [+0.54, +6.67] | 1 | Logs |
| ➖ | quality_gate_logs | % cpu utilization | +2.40 | [+0.77, +4.04] | 1 | Logs bounds checks dashboard |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.41 | [+0.27, +0.54] | 1 | Logs |
| ➖ | ddot_metrics_sum_cumulativetodelta_exporter | memory utilization | +0.34 | [+0.11, +0.56] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.19 | [+0.14, +0.24] | 1 | Logs |
| ➖ | ddot_metrics | memory utilization | +0.14 | [-0.03, +0.31] | 1 | Logs |
| ➖ | docker_containers_memory | memory utilization | +0.13 | [+0.06, +0.20] | 1 | Logs |
| ➖ | quality_gate_metrics_logs | memory utilization | +0.11 | [-0.12, +0.34] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.05 | [+0.00, +0.10] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | +0.02 | [-0.47, +0.51] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.01 | [-0.10, +0.11] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_v3 | ingress throughput | +0.00 | [-0.20, +0.20] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.20, +0.20] | 1 | Logs |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.03 | [-0.45, +0.39] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | -0.03 | [-0.43, +0.36] | 1 | Logs |
| ➖ | uds_dogstatsd_20mb_12k_contexts_20_senders | memory utilization | -0.04 | [-0.10, +0.02] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.04 | [-0.14, +0.05] | 1 | Logs |
| ➖ | otlp_ingest_logs | memory utilization | -0.07 | [-0.18, +0.03] | 1 | Logs |
| ➖ | ddot_metrics_sum_delta | memory utilization | -0.29 | [-0.47, -0.12] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.30 | [-0.34, -0.27] | 1 | Logs bounds checks dashboard |
| ➖ | ddot_metrics_sum_cumulative | memory utilization | -0.48 | [-0.62, -0.34] | 1 | Logs |
| ➖ | otlp_ingest_metrics | memory utilization | -0.57 | [-0.73, -0.41] | 1 | Logs |
| ➖ | ddot_logs | memory utilization | -0.58 | [-0.64, -0.52] | 1 | Logs |
Bounds Checks: ✅ Passed
| perf | experiment | bounds_check_name | replicates_passed | observed_value | links |
|---|---|---|---|---|---|
| ✅ | docker_containers_cpu | simple_check_run | 10/10 | 548 ≥ 26 | |
| ✅ | docker_containers_memory | memory_usage | 10/10 | 277.06MiB ≤ 370MiB | |
| ✅ | docker_containers_memory | simple_check_run | 10/10 | 709 ≥ 26 | |
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 | 0.19GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_0ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 | 0.24GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_1000ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 | 0.20GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_100ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 | 0.21GiB ≤ 1.20GiB | |
| ✅ | file_to_blackhole_500ms_latency | missed_bytes | 10/10 | 0B = 0B | |
| ✅ | quality_gate_idle | intake_connections | 10/10 | 3 = 3 | bounds checks dashboard |
| ✅ | quality_gate_idle | memory_usage | 10/10 | 174.41MiB ≤ 175MiB | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | intake_connections | 10/10 | 3 = 3 | bounds checks dashboard |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 | 507.76MiB ≤ 550MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | intake_connections | 10/10 | 3 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_logs | memory_usage | 10/10 | 205.00MiB ≤ 220MiB | bounds checks dashboard |
| ✅ | quality_gate_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | cpu_usage | 10/10 | 356.79 ≤ 2000 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | intake_connections | 10/10 | 4 ≤ 6 | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | memory_usage | 10/10 | 427.62MiB ≤ 475MiB | bounds checks dashboard |
| ✅ | quality_gate_metrics_logs | missed_bytes | 10/10 | 0B = 0B | bounds checks dashboard |
Explanation
Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
CI Pass/Fail Decision
✅ Passed. All Quality Gates passed.
- quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
- quality_gate_metrics_logs, bounds check missed_bytes: 10/10 replicas passed. Gate passed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds support for reading span/trace IDs from native applications using the OpenTelemetry Thread Local Context Record format (OTel spec PR #4947) in the CWS eBPF security module.
The implementation adds a new eBPF read path that:
fsbase) fromtask_struct->thread.fsbasevalid == 1) and converts W3C byte-order trace-id/span-id to the existing native-endianspan_context_tformatThe existing Datadog proprietary span TLS mechanism is preserved —
fill_span_context()tries the Datadog path first, then falls back to OTel. All downstream code (event structs, unmarshallers, hooks) is unchanged since the output remains the same24-byte
span_context_t.This targets native applications using ELF TLSDESC (C, C++, Rust, Java/JNI, .NET/FFI, Python native extensions). Support for additional runtimes (Go via pprof labels, Node.js via runtime internals) will be added separately. Currently x86_64 only
— ARM64 (
tpidr_el0) is deferred.Motivation
The OpenTelemetry specification defines a standard mechanism for SDKs to publish thread-level trace context (trace ID, span ID) through ELF Thread Local Storage so that out-of-process readers like eBPF probes can correlate security events with
distributed traces. Supporting this standard allows CWS to automatically pick up span context from any OTel-instrumented native application without requiring Datadog-specific instrumentation.
Describe how you validated your changes
probe/erpc,probe/constantfetch,ebpf/probes,ptracer,probe) build cleanlyotel-span-opencommandTestOTelSpan) that:__threadTLS variable in C, computes its offset fromfsbaseviaarch_prctl(ARCH_GET_FS)REGISTER_OTEL_TLS_OPeRPC operationopen()syscall and verifies the correct trace/span IDs appear in the captured security eventTestSpantest continues to work (Datadog proprietary path is unchanged)Additional Notes
task_struct->thread.fsbaseoffset is resolved using two separate BTF lookups (task_struct.thread+thread_struct.fsbase) because the BTF constant fetcher only recurses into anonymous struct members, andthreadis a named memberErrorSentinel— BTF is strongly preferred for these offsets sincetask_struct.threadvaries significantly with kernel configattrs-datafield (custom attributes) is not read — only trace-id and span-id are extractedfill_span_context()behavior has a minor semantic change: if the Datadog TLS is registered but returns all-zero context (no active span), it now falls through to try OTel before returning zeros