Merged
Conversation
## Which issue does this PR close? - Closes #19104. ## Rationale for this change #19104 explains but some datetime functions respect `execution.time_zone` config but there are no SQL examples for them. This PR adds it and also missing sql example for `date_part`, `extract` and `date_trunc` ## What changes are included in this PR? sql example changes and generated doc ## Are these changes tested? ## Are there any user-facing changes?
## Which issue does this PR close? - Closes #10048. ## Rationale for this change Lateral joins are a commonly used SQL feature that allows the right-side join relation to access columns from the left-side of the join. Like correlated subqueries, two popular evaluation strategies are nested loops (re-evaluate the right-side of the join for each row of the left join input) and decorrelation (rewrite the right join input to remove the correlation, converting the lateral join into a standard join with the correlation predicates as join conditions). Decorrelation is typically much faster because the right side is evaluated once rather than re-executed for every row of the left input. Previously, DataFusion had some support for evaluating lateral joins via decorrelation, but it was not functional. This PR fixes and extends the existing code to make basic lateral joins functional, although several notable TODOs remain. This PR also adds a suite of SLT tests for lateral joins (derived from the DuckDB and Postgres tests), covering both implemented and to-be-implemented behavior. Remaining extensions: * LATERAL subqueries with HAVING clauses (#21198) * LEFT JOIN LATERAL (#21199) * LATERAL subqueries with outer relation references outside the WHERE clause (#21201) ## What changes are included in this PR? * Match query structure properly (unwrap `SubqueryAlias`) so that lateral subqueries are recognized properly, even if they have aliases * Handle nested LATERAL clauses; each LATERAL can only reference sibling outer relations * Properly handle "the count bug", following similar logic to what we do for this case with correlated subqueries * Remove a `todo!` panic in the physical planner if a `Subquery` node is seen; these just represent a subquery structure we aren't able to decorrelate yet * Properly raise an error and bail out for LATERAL subqueries with HAVING clauses * Add SLT test suite for lateral joins (~33 queries), based in part on DuckDB and Postgres test suites * Update expected EXPLAIN output in various places * Add docs for `LATERAL JOIN` ## Are these changes tested? Yes; new tests added. I ran the test suite against DuckDB and confirmed that we get the same results under DuckDB, except for cases where DuckDB supports a class of lateral joins we don't support yet. ## Are there any user-facing changes? Yes; lateral joins now work for a wide swath of useful scenarios.
Bumps [object_store](https://github.com/apache/arrow-rs-object-store) from 0.13.1 to 0.13.2. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/apache/arrow-rs-object-store/blob/main/CHANGELOG-old.md">object_store's changelog</a>.</em></p> <blockquote> <!-- raw HTML omitted --> <h1>Historical Changelog</h1> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/7a65b75b0d26fd8a282999462cb7030fb85fdcc3"><code>7a65b75</code></a> chore: prepare <code>0.13.2</code> release (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/671">#671</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/d22f7f91d576ab7b2a12750751a76d30018cb12d"><code>d22f7f9</code></a> fix: missing 5xx error body when retry exhausted (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/618">#618</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/ebaee78c8bd05235b4ceb2a5125a125faae5da86"><code>ebaee78</code></a> Support --xa-s3 suffix for S3 Express One Zone bucket access points (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/663">#663</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/7ded938312299ccc1ca517e34fe34326d8426ebb"><code>7ded938</code></a> Replace <code>Path::child</code> with <code>Path::join</code> (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/666">#666</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/bdcac43fc637ac89cc80f18f4e16b3fb66fd0ec8"><code>bdcac43</code></a> feat: Add support for AWS_ENDPOINT_URL_S3 environment variable (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/590">#590</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/ca15c63f652e1c5c2e776231ae77d08bb46375ea"><code>ca15c63</code></a> Implement Clone for local and memory stores (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/653">#653</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/41d32428cd625e4b86d3411b4f06a0dff7ecab47"><code>41d3242</code></a> docs: clarify <code>Clone</code> behavior (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/656">#656</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/62592027cb7d33ed843bc14fb77d7fc32f13ace5"><code>6259202</code></a> Unify <code>from_env</code> behaviours (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/652">#652</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/66e640d1ba7b79d878e9423924ed5bff4f321b75"><code>66e640d</code></a> Switch TokenCache to RWLock (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/648">#648</a>)</li> <li><a href="https://github.com/apache/arrow-rs-object-store/commit/907653e46232edb3a635969bf288324b68d62cdc"><code>907653e</code></a> docs: add examples to the aws docs where appropriate (<a href="https://redirect.github.com/apache/arrow-rs-object-store/issues/651">#651</a>)</li> <li>Additional commits viewable in <a href="https://github.com/apache/arrow-rs-object-store/compare/v0.13.1...v0.13.2">compare view</a></li> </ul> </details> <br /> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Oleks V <comphead@users.noreply.github.com> Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>
) Related to #6880 Follow up on #17538 Bringing back the merge queues 🤞🏻 Last time it was painful because: - there was no `.asf.yaml` validation - if something goes wrong, we weren't able to force merge without the infra team this PR solved the first problem. As for the second one, I have the permissions to bypass so should be able to quickly revert if something ever goes bad <img width="830" height="151" alt="image" src="https://github.com/user-attachments/assets/62fb2bb5-44d8-4aae-a006-188d96728140" /> Check result: https://github.com/apache/datafusion/actions/runs/23715604583/job/69082077684?pr=21239 I also checked that CI will keep working by [merging](apache/datafusion-sandbox#197) this into our sandbox and then opening and merging [a dummy pr](apache/datafusion-sandbox#204) For now, bringing just the basic checks (in dev.yml). Will do rust.yml separately if everything goes smoothly after this one is merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )