Data Product Observability with OpenTelemetry #74
Description
Which problem is this feature request solving?
For data products, adding a telemetry/observability layer isn’t just a technical embellishment—it directly addresses the core challenges of turning data into a reliable, trusted product.
- Builds trust and reliability
- Provides measurable performance metrics
- Enables real‑time anomaly detection and faster resolution
- Improves efficiency and frees capacity
- Supports compliance and governance
- Delivers a unified view across the data lifecycle
In short, embedding telemetry into data products turns raw pipelines into observable products—ones that can be monitored, measured and trusted. The result is higher data reliability, faster issue resolution, stronger compliance, and clearer demonstration of value to stakeholders
Describe the solution you'd like
Placing observability/telemetry under dataAccess is the most sensible choice in ODPS 4.0 for a few reasons:
The dataAccess object is already the section that describes how users or machines can technically access a data product. Each entry (e.g. default, API, Agent) represents a distinct interface with its own metadata, authentication and documentation
Telemetry endpoints (metrics, traces, logs) are tightly coupled to these interfaces – they tell consumers how the API performs or how to collect monitoring data. Extending each access method to include observability metadata (instrumentation standard, metrics endpoint, traces endpoint, etc.) keeps the description of an interface and its monitoring in one place.
Data Quality in ODPS focuses on the content of the data (schema correctness, accuracy, completeness, etc.), while observability concerns the behaviour of the service delivering that data. Likewise, the SLA section is meant to capture contractual objectives like uptime and response time; it doesn’t describe how to collect metrics or traces. Adding telemetry under dataAccess avoids conflating content quality or contractual objectives with instrumentation.
Other data-product specifications follow a similar pattern. In the Data Mesh Data Product Descriptor specification, an “observability port” is defined as an interface that exposes logs, traces and metrics
ODPS doesn’t currently have a dedicated observability port, but extending each dataAccess entry with a telemetry block mirrors this idea and keeps ODPS aligned with broader industry practice.
Describe alternatives you've considered
None at the moment
Can you submit a pull request?
Yes.
``