Fix sleep function tests #4
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Modified test to return both cpu_time and validation result so the actual cpu_time value is visible in test output when validation fails. This helps debug timing issues in the sleep function tests.
krleonid
pushed a commit
that referenced
this pull request
Dec 30, 2025
This PR improves the `CommonSubplanOptimizer`, and should put it in
"maintenance mode", at least for now, and I think I'm done improving it
for a while.
## Multiple Nested Matching
This PR allows multiple subplans to be matched, rather than just a
single subplan, and allows nesting. Take the following query, for
example:
```sql
explain
select distinct range from range(10)
union all
select distinct range from range(10)
union all
select range % 2 as range from (select distinct range from range(10)) group by range
union all
select range % 2 as range from (select distinct range from range(10)) group by range
union all
select count(*) from (select range % 2 as range from (select distinct range from range(10)) group by range)
union all
select count(*) from (select range % 2 as range from (select distinct range from range(10)) group by range);
```
This query unions 6 queries together. Each query occurs twice, and is
nested in the next query. With the optimizer disabled, this yields the
following plan:
```
┌───────────────────────────┐
│ UNION ├──────────────┬────────────────────────────┬────────────────────────────┬────────────────────────────┬────────────────────────────┐
└─────────────┬─────────────┘ │ │ │ │ │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ HASH_GROUP_BY ││ HASH_GROUP_BY ││ PROJECTION ││ PROJECTION ││ UNGROUPED_AGGREGATE ││ UNGROUPED_AGGREGATE │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ Groups: #0 ││ Groups: #0 ││ range ││ range ││ Aggregates: ││ Aggregates: │
│ ││ ││ ││ ││ count_star() ││ count_star() │
│ ~10 rows ││ ~10 rows ││ ~6 rows ││ ~6 rows ││ ││ │
└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ PROJECTION ││ PROJECTION ││ HASH_GROUP_BY ││ HASH_GROUP_BY ││ PROJECTION ││ PROJECTION │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ range ││ range ││ Groups: #0 ││ Groups: #0 ││ 42 ││ 42 │
│ ││ ││ ││ ││ ││ │
│ ~10 rows ││ ~10 rows ││ ~6 rows ││ ~6 rows ││ ~6 rows ││ ~6 rows │
└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ RANGE ││ RANGE ││ PROJECTION ││ PROJECTION ││ HASH_GROUP_BY ││ HASH_GROUP_BY │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ Function: RANGE ││ Function: RANGE ││ range ││ range ││ Groups: #0 ││ Groups: #0 │
│ ││ ││ ││ ││ ││ │
│ ~10 rows ││ ~10 rows ││ ~10 rows ││ ~10 rows ││ ~6 rows ││ ~6 rows │
└───────────────────────────┘└───────────────────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ HASH_GROUP_BY ││ HASH_GROUP_BY ││ PROJECTION ││ PROJECTION │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ Groups: #0 ││ Groups: #0 ││ range ││ range │
│ ││ ││ ││ │
│ ~10 rows ││ ~10 rows ││ ~10 rows ││ ~10 rows │
└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ PROJECTION ││ PROJECTION ││ HASH_GROUP_BY ││ HASH_GROUP_BY │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ range ││ range ││ Groups: #0 ││ Groups: #0 │
│ ││ ││ ││ │
│ ~10 rows ││ ~10 rows ││ ~10 rows ││ ~10 rows │
└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ RANGE ││ RANGE ││ PROJECTION ││ PROJECTION │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ Function: RANGE ││ Function: RANGE ││ range ││ range │
│ ││ ││ ││ │
│ ~10 rows ││ ~10 rows ││ ~10 rows ││ ~10 rows │
└───────────────────────────┘└───────────────────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ RANGE ││ RANGE │
│ ──────────────────── ││ ──────────────────── │
│ Function: RANGE ││ Function: RANGE │
│ ││ │
│ ~10 rows ││ ~10 rows │
└───────────────────────────┘└───────────────────────────┘
```
As we can see, there is a lot of redundance here. With this PR (and the
optimizer enabled, of course), we get the following plan:
```
┌───────────────────────┐
│ CTE │
│ ──────────────── │
│ CTE Name: │
│ __common_subplan_1 │
│ ├────────────┐
│ Table Index: 79 │ │
│ │ │
│ ~0 rows │ │
└───────────┬───────────┘ │
┌───────────┴───────────┐┌───────────┴───────────┐
│ HASH_GROUP_BY ││ CTE │
│ ──────────────── ││ ──────────────── │
│ Groups: #0 ││ CTE Name: │
│ ││ __common_subplan_2 │
│ ││ ├────────────┐
│ ││ Table Index: 86 │ │
│ ││ │ │
│ ~10 rows ││ ~0 rows │ │
└───────────┬───────────┘└───────────┬───────────┘ │
┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐
│ PROJECTION ││ HASH_GROUP_BY ││ CTE │
│ ──────────────── ││ ──────────────── ││ ──────────────── │
│ range ││ Groups: #0 ││ CTE Name: │
│ ││ ││ __common_subplan_3 │
│ ││ ││ ├────────────┐
│ ││ ││ Table Index: 91 │ │
│ ││ ││ │ │
│ ~10 rows ││ ~6 rows ││ ~0 rows │ │
└───────────┬───────────┘└───────────┬───────────┘└───────────┬───────────┘ │
┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐
│ RANGE ││ PROJECTION ││ UNGROUPED_AGGREGATE ││ UNION │
│ ──────────────── ││ ──────────────── ││ ──────────────── ││ │
│ Function: RANGE ││ range ││ Aggregates: ││ ├────────────┬────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────┐
│ ││ ││ count_star() ││ │ │ │ │ │ │
│ ~10 rows ││ ~10 rows ││ ││ │ │ │ │ │ │
└───────────────────────┘└───────────┬───────────┘└───────────┬───────────┘└───────────┬───────────┘ │ │ │ │ │
┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐┌───────────┴───────────┐
│ CTE_SCAN ││ PROJECTION ││ CTE_SCAN ││ CTE_SCAN ││ PROJECTION ││ PROJECTION ││ CTE_SCAN ││ CTE_SCAN │
│ ──────────────── ││ ──────────────── ││ ──────────────── ││ ──────────────── ││ ──────────────── ││ ──────────────── ││ ──────────────── ││ ──────────────── │
│ CTE Index: 79 ││ 42 ││ CTE Index: 79 ││ CTE Index: 79 ││ range ││ range ││ CTE Index: 91 ││ CTE Index: 91 │
│ ││ ││ ││ ││ ││ ││ ││ │
│ ~10 rows ││ ~6 rows ││ ~10 rows ││ ~10 rows ││ ~6 rows ││ ~6 rows ││ ~1 row ││ ~1 row │
└───────────────────────┘└───────────┬───────────┘└───────────────────────┘└───────────────────────┘└───────────┬───────────┘└───────────┬───────────┘└───────────────────────┘└───────────────────────┘
┌───────────┴───────────┐ ┌───────────┴───────────┐┌───────────┴───────────┐
│ CTE_SCAN │ │ CTE_SCAN ││ CTE_SCAN │
│ ──────────────── │ │ ──────────────── ││ ──────────────── │
│ CTE Index: 86 │ │ CTE Index: 86 ││ CTE Index: 86 │
│ │ │ ││ │
│ ~6 rows │ │ ~6 rows ││ ~6 rows │
└───────────────────────┘ └───────────────────────┘└───────────────────────┘
```
Which has 3 CTEs, 8 CTE scans, and only 3 aggregations (instead of the
original 12!).
## Fuzzy Plan Matching
Something that can show up in query plans is an "almost" exact subplan
match. Before this PR, an exact match was required. With this PR, we can
do "fuzzy" matching, where the plan is mostly the same, save for some
selected columns. If we have the following query, for example:
```sql
-- Create build table
create table build as
select range a, range * 2 b, range * 3 c, range * 4 d
from (select range::utinyint as range from range(11))
where range % 5 = 0;
-- Create probe table
create table probe as
select range e, range * 2 f, range * 3 g, range * 4 h
from (select range::utinyint as range from range(11));
-- View that joins the two
create view my_view as
from probe join build on (build.a = probe.e);
-- Select greatest of all columns, unioned with greatest of just two columns
explain
select greatest(a, b, c, d, e, f, g, h) from my_view
union all
select greatest(b, f) from my_view;
```
Here, one of the unioned queries selects all columns, and the other
selects just two columns. As we can see, the join (coming from the view)
in the second query is "contained" in the join in first query, as the
first query selects all columns that the second query needs.
We currently get the following plan (the optimizer doesn't trigger):
```
┌───────────────────────────┐
│ UNION ├───────────────────────────────────────────┐
└─────────────┬─────────────┘ │
┌─────────────┴─────────────┐ ┌─────────────┴─────────────┐
│ PROJECTION │ │ PROJECTION │
│ ──────────────────── │ │ ──────────────────── │
│ greatest(a, b, c, d, e, f,│ │ greatest(b, f) │
│ g, h) │ │ │
│ │ │ │
│ ~3 rows │ │ ~3 rows │
└─────────────┬─────────────┘ └─────────────┬─────────────┘
┌─────────────┴─────────────┐ ┌─────────────┴─────────────┐
│ PROJECTION │ │ HASH_JOIN │
│ ──────────────────── │ │ ──────────────────── │
│ e │ │ Join Type: INNER │
│ f │ │ Conditions: e = a │
│ g │ │ │
│ h │ │ │
│ a │ │ ├──────────────┐
│ b │ │ │ │
│ c │ │ │ │
│ d │ │ │ │
│ │ │ │ │
│ ~3 rows │ │ ~3 rows │ │
└─────────────┬─────────────┘ └─────────────┬─────────────┘ │
┌─────────────┴─────────────┐ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ HASH_JOIN │ │ SEQ_SCAN ││ SEQ_SCAN │
│ ──────────────────── │ │ ──────────────────── ││ ──────────────────── │
│ Join Type: INNER │ │ Table: probe ││ Table: build │
│ Conditions: e = a │ │ Type: Sequential Scan ││ Type: Sequential Scan │
│ │ │ ││ │
│ ├──────────────┐ │ Projections: ││ Projections: │
│ │ │ │ e ││ a │
│ │ │ │ f ││ b │
│ │ │ │ ││ │
│ ~3 rows │ │ │ ~11 rows ││ ~3 rows │
└─────────────┬─────────────┘ │ └───────────────────────────┘└───────────────────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ SEQ_SCAN ││ SEQ_SCAN │
│ ──────────────────── ││ ──────────────────── │
│ Table: probe ││ Table: build │
│ Type: Sequential Scan ││ Type: Sequential Scan │
│ ││ │
│ Projections: ││ Projections: │
│ e ││ a │
│ f ││ b │
│ g ││ c │
│ h ││ d │
│ ││ │
│ ~11 rows ││ ~3 rows │
└───────────────────────────┘└───────────────────────────┘
```
With the improvements to the optimizer in this PR, we now get this plan:
```
┌───────────────────────────┐
│ CTE │
│ ──────────────────── │
│ CTE Name: │
│ __common_subplan_1 │
│ ├───────────────────────────────────────────┐
│ Table Index: 29 │ │
│ │ │
│ ~0 rows │ │
└─────────────┬─────────────┘ │
┌─────────────┴─────────────┐ ┌─────────────┴─────────────┐
│ HASH_JOIN │ │ UNION │
│ ──────────────────── │ │ │
│ Join Type: INNER │ │ │
│ Conditions: e = a ├──────────────┐ │ ├──────────────┐
│ │ │ │ │ │
│ ~3 rows │ │ │ │ │
└─────────────┬─────────────┘ │ └─────────────┬─────────────┘ │
┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ SEQ_SCAN ││ SEQ_SCAN ││ PROJECTION ││ PROJECTION │
│ ──────────────────── ││ ──────────────────── ││ ──────────────────── ││ ──────────────────── │
│ Table: probe ││ Table: build ││ greatest(a, b, c, d, e, f,││ greatest(b, f) │
│ Type: Sequential Scan ││ Type: Sequential Scan ││ g, h) ││ │
│ ││ ││ ││ │
│ Projections: ││ Projections: ││ ││ │
│ e ││ a ││ ││ │
│ f ││ b ││ ││ │
│ g ││ c ││ ││ │
│ h ││ d ││ ││ │
│ ││ ││ ││ │
│ ~11 rows ││ ~3 rows ││ ~3 rows ││ ~3 rows │
└───────────────────────────┘└───────────────────────────┘└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ PROJECTION ││ PROJECTION │
│ ──────────────────── ││ ──────────────────── │
│ e ││ #1 │
│ f ││ #4 │
│ g ││ │
│ h ││ │
│ a ││ │
│ b ││ │
│ c ││ │
│ d ││ │
│ ││ │
│ ~3 rows ││ ~0 rows │
└─────────────┬─────────────┘└─────────────┬─────────────┘
┌─────────────┴─────────────┐┌─────────────┴─────────────┐
│ CTE_SCAN ││ CTE_SCAN │
│ ──────────────────── ││ ──────────────────── │
│ CTE Index: 29 ││ CTE Index: 29 │
│ ││ │
│ ~3 rows ││ ~3 rows │
└───────────────────────────┘└───────────────────────────┘
```
As we can see, the join is materialized as a CTE, and scanned twice.
Columns that aren't needed are projected out after the CTE scan.
## Benchmark Improvements
With the changes, we now trigger more subplan elimination on TPC-DS and
TPC-H. Here are some results that I collected on my laptop.
TPC-DS SF100 improvements:
Q61: 0.61s -> 0.33s (~1.8x)
Q70: 0.81s -> 0.55s (~1.5x)
TPC-H SF100 improvements:
Q11: 0.16s -> 0.12s (~1.3x)
## Notes
I needed a lot of indirection to get the column bindings to match with
the fuzzy plan matching, which required a lot of `unordered_map`s. To
avoid doing so many allocations, I've also implemented
`arena_unordered_map`, so that the number of allocations grow
logarithmically. I'm not sure how much this helped, but it's just
something I wanted to do as we are trying to reduce allocations.
Overall, this optimizer now takes ~10% of total optimization time.
krleonid
pushed a commit
that referenced
this pull request
Jan 14, 2026
…0283) Fix for: duckdblabs/duckdb-internal#6809 , duckdb#20086 I would like someone to take a look at this before I run CI, to see if the fix makes sense. In ConstantOrNullFunction, there is a bug where if the first loop iteration is a FLAT_VECTOR, the result validity mask is created as a reference to the validity mask of args.data[idx]. If the subsequent iteration is the default branch (say, a DICTIONARY_VECTOR), and we call result_mask.SetInvalid(i), this is now overwriting the validity mask of the first input column where the reference was created. I believe the fix for this is to call EnsureWritable in the FLAT_VECTOR case, to make sure the validity mask is not a reference to the input's validity mask before we call ```cpp result_mask.Combine(input_mask, args.size()) ``` (which is where the alias is actually created). The reproducer hits this case -- a specific scenario of unique index + update + no checkpointing was leading to the this scenario. For reference, here is the query plan of the last query in the reproducer, where the bug was occuring. The t1.c0 column is being passed as a FLAT_VECTOR to constantOrNullFunction, and the t0.c1 column is being passed in as a dictionary vector. Since the argument at index 1 in ConstantOrNullFunction is the c0 column in the output, we were overwriting NULLs into the ouput since the filter was overwriting the validity mask in ConstantOrNullFunction: ``` ┌───────┬───────┬───────┐ │ c0 │ c0 │ c1 │ │ int32 │ int32 │ int32 │ ├───────┼───────┼───────┤ │ NULL │ 1 │ NULL │ │ NULL │ -1 │ NULL │ └───────┴───────┴───────┘ ``` Whereas it should be: ``` ┌───────┬───────┬───────┐ │ c0 │ c0 │ c1 │ │ int32 │ int32 │ int32 │ ├───────┼───────┼───────┤ │ 0 │ 1 │ NULL │ │ NULL │ -1 │ NULL │ └───────┴───────┴───────┘ ``` ```┌────────────────────────────────────────────────┐ │┌──────────────────────────────────────────────┐│ ││ Total Time: 9.18s ││ │└──────────────────────────────────────────────┘│ └────────────────────────────────────────────────┘ ┌───────────────────────────┐ │ QUERY │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ EXPLAIN_ANALYZE │ │ ──────────────────── │ │ 0 rows │ │ (0.00s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ c0 │ │ c0 │ │ c1 │ │ │ │ 2 rows │ │ (0.00s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ #3 │ │ #7 │ │ #11 │ │ │ │ 2 rows │ │ (0.00s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ FILTER │ │ ──────────────────── │ │ (constant_or_null(false, │ │ c0, c1) IS NULL) │ │ │ │ 2 rows │ │ (1.82s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ NULL │ │ #6 │ │ NULL │ │ #5 │ │ NULL │ │ #4 │ │ NULL │ │ #3 │ │ NULL │ │ #2 │ │ NULL │ │ #1 │ │ NULL │ │ #0 │ │ NULL │ │ │ │ 2 rows │ │ (0.00s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ PROJECTION │ │ ──────────────────── │ │ NULL │ │ #2 │ │ NULL │ │ #1 │ │ NULL │ │ #0 │ │ NULL │ │ │ │ 2 rows │ │ (0.00s) │ └─────────────┬─────────────┘ ┌─────────────┴─────────────┐ │ POSITIONAL_SCAN │ │ ──────────────────── │ │ 2 rows ├──────────────┐ │ (7.30s) │ │ └─────────────┬─────────────┘ │ ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ │ TABLE_SCAN ││ TABLE_SCAN │ │ ──────────────────── ││ ──────────────────── │ │ Table: t1 ││ Table: t0 │ │ Type: Sequential Scan ││ Type: Sequential Scan │ │ Projections: c0 ││ │ │ ││ Projections: │ │ ││ c1 │ │ ││ c0 │ │ ││ │ │ 0 rows ││ 0 rows │ │ (0.00s) ││ (0.00s) │ └───────────────────────────┘└───────────────────────────┘ ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.