[Feature Request]: Integrate Apache Beam with Open Lineage

[rohitsinha54]

What would you like to happen?

See 33980 for previous background around tracking lineage in Apache Beam direct runner.

Building up open the "Sending this information to some Lineage graph visualization tool."

Beam can report this lineage information in OpenLineage format. This allows Beam users to visualize the lineage with already supported lineage visualization tools of open lineage.

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[rohitsinha54]

.set-labels gsoc, gsoc2025

[github-actions]

Label cannot be managed because it does not exist in the repo. Please check your spelling.

[Strikerrx01]

.take-issue

[Strikerrx01]

Proposed Solution for OpenLineage Integration

I'd like to implement the integration between Apache Beam and OpenLineage.

Technical Approach

1. Create an OpenLineage Client for Beam

   • Generate OpenLineage-compatible events
   • Send events to OpenLineage API endpoints
   • Support configurable transport options

2. Implement Lineage Collection in Beam's Core

   • Extend PTransform and PCollection to capture metadata
   • Intercept data flow to track lineage
   • Add lineage context propagation

3. Design Integration Points

   • Pipeline options for OpenLineage configuration
   • Custom facets for Beam-specific metadata
   • Integration with existing metrics system

4. Support Multiple SDKs

   • Java implementation (core)
   • Python SDK wrapper
   • Consistent configuration across SDKs

Implementation Plan

• Exploration (2 weeks): Research and design
• Core Implementation (4 weeks): Client and tracking mechanism
• SDK Support (3 weeks): Java and Python implementations
• Testing & Documentation (3 weeks): Tests, examples, docs

Example Usage (Python)

import apache_beam as beam
from apache_beam.options.pipeline_options import PipelineOptions

options = PipelineOptions([
'--openlineage_enabled=true',
'--openlineage_url=http://localhost:5000',
'--openlineage_namespace=my_namespace',
'--openlineage_job_name=my_beam_job'
])

with beam.Pipeline(options=options) as p:
# Pipeline definition
# Lineage tracking will happen automatically

Challenges and Mitigations

• Cross-language support: Common Java core with language wrappers
• Runner differences: Abstraction layer for different runners
• Performance impact: Optimize for minimal overhead

I've studied the Apache Foundation GSoC guidelines and Beam's contribution process. I'm excited to contribute to this feature that will help users better understand and visualize data lineage.

Looking forward to your feedback!

[github-actions]

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.