Skip to content

feat: Add cs.remaining() selector#27301

Open
gab23r wants to merge 3 commits intopola-rs:mainfrom
gab23r:cs.remaining
Open

feat: Add cs.remaining() selector#27301
gab23r wants to merge 3 commits intopola-rs:mainfrom
gab23r:cs.remaining

Conversation

@gab23r
Copy link
Copy Markdown
Contributor

@gab23r gab23r commented Apr 14, 2026

Towards #12067

Add cs.remaining() selector that selects columns whose output names are not produced by other expressions in the same select() or with_columns() call.

Example:

import polars as pl
import polars.selectors as cs

df = pl.DataFrame({ "b": [4, 5, 6], "c": [7, 8, 9], "a": [1, 2, 3], "d": [10, 11, 12]})

# Reorder columns
df.select("a", cs.remaining(), "d")
# shape: (3, 4)
# ┌─────┬─────┬─────┬─────┐
# │ a   ┆ b   ┆ c   ┆ d   │
# │ --- ┆ --- ┆ --- ┆ --- │
# │ i64 ┆ i64 ┆ i64 ┆ i64 │
# ╞═════╪═════╪═════╪═════╡
# │ 1   ┆ 4   ┆ 7   ┆ 10  │
# │ 2   ┆ 5   ┆ 8   ┆ 11  │
# │ 3   ┆ 6   ┆ 9   ┆ 12  │
# └─────┴─────┴─────┴─────┘

The implementation uses a two-pass approach in rewrite_projections:

  1. Expand all non-Remaining expressions and collect their output column names
  2. Transform Selector::RemainingSelector::Wildcard - Selector::ByName(output_names)

I used AI to explore different implementations and I confirm that I have reviewed all changes myself, and I believe they are relevant and correct.

@github-actions github-actions bot added A-selectors Area: column selectors enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Apr 14, 2026
@gab23r
Copy link
Copy Markdown
Contributor Author

gab23r commented Apr 14, 2026

@alexander-beedie, curious about your thoughts as you already think about adding this selectors !

@github-actions github-actions bot added the changes-dsl Do not merge if this label is present and red. label Apr 14, 2026
@coastalwhite
Copy link
Copy Markdown
Collaborator

I would personally be quite uncomfortable with such a two-phase approach specifically for this selector. Like, how would this work at other places that are not with_columns or select? The two-phase approach was how selectors used to be implemented and caused a large amount of messiness, unpredictability, and just general bugginess.

For group_by(...).agg(...), cs.all() already doesn't include all columns. Here it is implemented through the ignored_selector_columns. Maybe such a mechanism can be altered or reused somehow?

@gab23r
Copy link
Copy Markdown
Contributor Author

gab23r commented Apr 14, 2026

I battle test it more and you were right, I had some bugs with selectors athimetics, I fixed them but now the logic starts to be quite messy... I will try to revisit using the already existing ignored_selector_columns

@github-actions github-actions bot removed the changes-dsl Do not merge if this label is present and red. label Apr 14, 2026
@gab23r
Copy link
Copy Markdown
Contributor Author

gab23r commented Apr 14, 2026

I now use the already existing ignored_selector_columns which is much better than doing this transformation: RemainingWildcard - ByName(output_columns).

@github-actions github-actions bot added the changes-dsl Do not merge if this label is present and red. label Apr 14, 2026
@gab23r
Copy link
Copy Markdown
Contributor Author

gab23r commented Apr 15, 2026

@coastalwhite, maybe you wanted the logic to happen before rewrite_projections like in the group_by case.
In this cs.remaining case, I don't see how to do that without modify a lot a logic a bit of everywhere. And I still think that applying the logic here make more sense.

I have battle test the implemention with other context than select or with_columns and I don't see issue and I can't imagine any potential futures bugs. But maybe you have more precise ideas where bug can show up here ?

I have added the unstable tag to cs.remaining.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-selectors Area: column selectors changes-dsl Do not merge if this label is present and red. enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants