Struct projection pushdown v2 #148

Tishj · 2025-11-28T12:32:06Z

This PR changes the struct projection pushdown to push the extract down to the storage.
I'll explain the differences in the different levels:

Current Behavior

Expression:

The struct_extract(s, 'a') expression remains unchanged

Storage:

We push down scan_child_column booleans, to indicate which fields are left unread from the STRUCT.
The fields that aren't touched are left empty (null) so we skip reading them.
What is emitted from the storage is a column of type STRUCT.

Execution:

The execution of the struct_extract function takes out the field it needs.

New Behavior

Expression:

The struct_extract(s, 'a') is replaced with a new BoundColumnRefExpression, referencing the field we emit directly from the LogicalGet

Storage:

We don't emit a STRUCT column if only a subset of the fields are needed by the query, instead we emit the individually referenced field(s)
What is emitted from the storage is now a column of the type of the field.

Execution:

The BoundReferenceExpression just references the field that was emitted directly by the storage.

Technical details

We communicate this new "pushdown extract" behavior through the ColumnIndex down to the StorageIndex, as well as the LogicalType of the field we're scanning.

Limitations

We disable the pushdown of the extract if fields of the struct are referenced by a filter.
To make that work correctly, we would need to rewrite the TableFilterSet, which I suspect has far-reaching consequences I'm not ready to deal with yet.

…ming it directly in the storage

…ushdown_v2

…Index as well

…ut not used yet

…rded in the transform to StorageIndex

src/function/table/table_scan.cpp

src/include/duckdb/planner/table_binding.hpp

src/include/duckdb/storage/table/scan_state.hpp

src/optimizer/remove_unused_columns.cpp

src/planner/operator/logical_get.cpp

src/storage/table/struct_column_data.cpp

…h it inside the StructColumnData explicitly, based on the StorageIndex we save in the ColumnScanState

…eFields method, handle the differing behavior through a callback

…ojected

…that requires lazy replacement of the expressions, instead of the eager replacement we're doing now

…n logic in a single location

…ushing down extract expressions

…call RewriteExpressions to break up the struct references into separate struct extracts

…leCatalogEntry

…ust distill the set from the 'child_columns' at the end

…nging the prototype of the entire FetchRow virtual method

… ColumnData::CheckZonemap

…ushdown_v2

…Initialize

…ushdown_v2

…ion_pushdown_v2

…we can just distill the set from the 'child_columns' at the end" This reverts commit 14240bf.

…ushdown_v2

e.g. https://github.com/duckdb/duckdb/actions/runs/20118040586/job/57733062113

…ushdown_v2

…tically be disabled if 'statistics_extended' is not available

Tishj added 7 commits November 6, 2025 16:47

clean up the existing code a bit

bf985c2

part 1: strip off the struct extract expression, as we will be perfor…

1327194

…ming it directly in the storage

first working state of pushing struct extract into storage

c5708c2

Merge remote-tracking branch 'upstream/main' into struct_projection_p…

5b6decd

…ushdown_v2

wip: adding type+logical_type to ColumnIndex, started work on Storage…

4f030e9

…Index as well

add the logical type + type to the StorageIndex as well, compiling, b…

d7abf5c

…ut not used yet

wip wip, pushing down the type of the field in the ColumnIndex, forwa…

01b8014

…rded in the transform to StorageIndex