perf: Create IR slice from expr slice pushdown

[nameexhaustion]

• Fixes Insert IR-level slice before select(all().head/tail(N)/first/last) #26592
• Depends on refactor(rust): Utility for identifying expr projection heights #27198

Create IR-level slices as the common slice of all expr slices that apply to columns. This can then be pushed down by the optimizer to perform the slice at lower levels.

In implementation, we add a new expr slice pushdown implementation via recursive tree traversal instead of going through OptimizationRule::optimize_expr, and this is called when doing IR slice pushdown on the exprs of each IR.

E.g. lf.select(pl.all().first()) can currently be much slower than lf.head(1), but after this PR it will optimize to the same as lf.head(1).

Detected exprs: head() / tail() / first() / last() / slice().

E.g. - The following now creates a slice at the scan:

print(pl.scan_csv(data).select(pl.first("a")).explain())
# SELECT [col("a").first()]
#   Csv SCAN [12 in-mem bytes]
#   PROJECT */1 COLUMNS
#   SLICE: Positive { offset: 0, len: 1 }
#   ESTIMATED ROWS: 6

E.g. - We are able to size the IR::Slice to the exact common boundary, then apply an offset correction to the expr slices -

print(
    pl.scan_csv(data)
    .select(a2=pl.col("a").slice(2, 1), a3=pl.col("a").slice(3, 1))
    .explain()
)
# SELECT [col("a").slice(offset=0, length=1).alias("a2"), col("a").slice(offset=1, length=1).alias("a3")]
#   Csv SCAN [12 in-mem bytes]
#   PROJECT */1 COLUMNS
#   SLICE: Positive { offset: 2, len: 2 }
#   ESTIMATED ROWS: 6

As well as for negative slices-

print(
    pl.scan_csv(data)
    .select(a2=pl.col("a").slice(-2, 1), a3=pl.col("a").slice(-3, 1))
    .explain()
)
# SELECT [col("a").slice(offset=-1, length=1).alias("a2"), col("a").slice(offset=-2, length=1).alias("a3")]
#   Csv SCAN [12 in-mem bytes]
#   PROJECT */1 COLUMNS
#   SLICE: Negative { offset_from_end: 3, len: 2 }
#   ESTIMATED ROWS: 6

--

• refactor(rust): Utility for identifying expr projection heights #27198
  • refactor(rust): Add generic tree traversal with edge value propagation #27249
    • > perf: Create IR slice from expr slice pushdown #27200

[github-actions]

The uncompressed lib size after this PR is 58.6537 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6615 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6620 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6692 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6603 MB.

[codecov]

Codecov Report

❌ Patch coverage is 95.75758% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.91%. Comparing base (c492bc0) to head (24d289f).

Files with missing lines	Patch %	Lines
...rs-plan/src/plans/optimizer/slice_pushdown_expr.rs	96.09%	11 Missing ⚠️
crates/polars-utils/src/collection.rs	75.00%	3 Missing ⚠️

Additional details and impacted files

@@                       Coverage Diff                        @@
##           nxs/expr-projection-height-2   #27200      +/-   ##
================================================================
+ Coverage                         81.88%   81.91%   +0.03%     
================================================================
  Files                              1823     1824       +1     
  Lines                            251386   251618     +232     
  Branches                           3149     3149              
================================================================
+ Hits                             205842   206109     +267     
+ Misses                            44736    44701      -35     
  Partials                            808      808              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

The uncompressed lib size after this PR is 58.6682 MB.

[github-actions]

The uncompressed lib size after this PR is 58.4668 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6704 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6730 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6718 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6717 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6603 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6753 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6753 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6753 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6746 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6767 MB.

[github-actions]

The uncompressed lib size after this PR is 58.8740 MB.

[github-actions]

The uncompressed lib size after this PR is 58.8732 MB.

[github-actions]

The uncompressed lib size after this PR is 58.8688 MB.

[github-actions]

The uncompressed lib size after this PR is 58.6718 MB.

[github-actions]

The uncompressed lib size after this PR is 61.4957 MB.

[github-actions]

The uncompressed lib size after this PR is 60.9760 MB.

[github-actions]

The uncompressed lib size after this PR is 60.6818 MB.