Overview
The FilterPhase step now produces data using Polars LazyFrames, but the ConcatFieldPhase step still uses the legacy implementation and cannot process this new output. To keep the pipeline consistent and efficient, the FilterPhase phase must be updated to use the Polars version. This will keep existing behaviour while modernising the underlying process.
Tech Approach
- Update digital_land/phase/FilterPhase.py so it delegates to the Polars FilterPhase implementation.
- Implement the full FilterPhase logic inside digital_land/phase_polars/transform/FilterPhase.py.
- Convert existing FilterPhase rules into Polars lazy transformations.
- Ensure the FilterPhase phase accepts the LazyFrame produced by the new FilterPhase step.
- Keep the same public interface so no downstream code needs changing.
Acceptance Criteria/Tests
Resourcing & Dependencies
Depends on ConcatFieldPhase, no external teams.
Overview
The FilterPhase step now produces data using Polars LazyFrames, but the ConcatFieldPhase step still uses the legacy implementation and cannot process this new output. To keep the pipeline consistent and efficient, the FilterPhase phase must be updated to use the Polars version. This will keep existing behaviour while modernising the underlying process.
Tech Approach
Acceptance Criteria/Tests
Resourcing & Dependencies
Depends on ConcatFieldPhase, no external teams.