Add EDOT data streams documentation and clarify exporter/storage behavior #504
Conversation
Vale Linting ResultsSummary: 19 suggestions found 💡 Suggestions (19)
|
🔍 Preview links for changed docs |
theletterf
left a comment
theletterf
left a comment
There was a problem hiding this comment.
Very useful doc! I suggest a review by @carsonip and @AlexanderWert
carsonip
left a comment
carsonip
left a comment
There was a problem hiding this comment.
May be good to have @felixbarny 's 👀 too
docs/reference/data-streams.md
Outdated
| ## Storage engines used by EDOT | ||
|
|
||
| | Data type | Storage engine | | ||
| |-----------|----------------| | ||
| | Metrics | TSDB (`mode: time_series`) | | ||
| | Logs | LogsDB | | ||
| | Traces | Standard time-series model | |
There was a problem hiding this comment.
I'm not sure what's the audience of this doc, but mapping EDOT to storage engine is not that simple. EDOT is responsible for sending to a data stream, and depending on which data stream it sends to, different index templates may apply, and the index templates decide what storage engine is used.
e.g. EDOT with otel mapping mode sends to otel data streams, and the table is correct. But EDOT with ecs mapping mode sending to apm data streams, the table will be incorrect.
There was a problem hiding this comment.
Ah, I should have known it wouldn't be that simple 😄 I decided to let go of the table altogether in favor of a more descriptive approach that hopefully captures the "it depends" reality better.
docs/reference/data-streams.md
Outdated
|
|
||
| ### ECS-compatible mode | ||
|
|
||
| When the `ecs` exporter is enabled, or when ingesting data from {{product.apm}} agents using `elasticapmintake`, EDOT writes to: |
There was a problem hiding this comment.
Does elasticapmintake decide the destination data stream?
There was a problem hiding this comment.
We're in the process of removing the mapping mode configuration and we should avoid mentioning the mapping mode, particularly the legacy ecs mapping mode in the configuration.
From Felix's comment, apparently yes
There was a problem hiding this comment.
At least, this is how we should do it. @gregkalapos to double-check if we're already doing it like that.
docs/reference/data-streams.md
Outdated
|
|
||
| To learn how to route OpenTelemetry (OTel) signals to custom data streams, refer to [Data stream routing](docs-content://solutions/observability/apm/opentelemetry/data-stream-routing.md). | ||
|
|
||
| ## Exporter behavior: `otel` and `ecs` |
There was a problem hiding this comment.
We're in the process of removing the mapping mode configuration and we should avoid mentioning the mapping mode, particularly the legacy ecs mapping mode in the configuration.
There's probably still a need to document the behavior of the elasticapmintake receiver, which internally instructs the exporter to use the ecs mode. But that's an implementation detail we shouldn't mention. What's relevant to the user is that the elasticapmintake is running in a compatibility mode where it produces documents similar to what APM server was doing. However, we should make clear that this is only intended for migration use cases and we should steer users towards using OTel SDKs.
There was a problem hiding this comment.
I changed the page to reflect that and made it clear that the ecs mode is for backwards compatibility only. If you think a more drastic change is warranted (e.g. removing any mention of the mapping modes), let me know.
|
@carsonip @felixbarny - does it look ok to you after the most recent changes? LMK if I can merge this! |
4 participants
Summary
This PR adds a new documentation page titled “EDOT data streams”, addressing recurring user questions.
The new page explains:
Why EDOT configurations sometimes include two exporters (
otelandecs)Which data streams are populated by EDOT in OTel-native and ECS/APM-compatible modes
Differences in data streams between mOTLP and local EDOT deployments
Why some fields appear duplicated
Which storage engines are used (TSDB, LogsDB)
Closes #3501
Generative AI disclosure
Did you use a generative AI (GenAI) tool to assist in creating this contribution?
If you answered "Yes" to the previous question, please specify the tool(s) and model(s) used (e.g., Google Gemini, OpenAI ChatGPT-4, etc.).
Tool(s) and model(s) used:
Claude 4.5 Sonnet via Cursor