Skip to content

[GLUTEN-7600][VL] Add monotonically_increasing_id function mapping #11674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
n0r0shi wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[GLUTEN-7600][VL] Add monotonically_increasing_id function mapping #11674

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions backends-velox/src/test/scala/org/apache/gluten/functions/ScalarFunctionsValidateSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -496,12 +496,17 @@ abstract class ScalarFunctionsValidateSuite extends FunctionsValidateSuite {
}
}

// FIXME: Ignored: https://github.com/apache/incubator-gluten/issues/7600.
ignore("monotonically_increasintestg_id") {
test("monotonically_increasing_id") {
runQueryAndCompare("""SELECT monotonically_increasing_id(), l_orderkey
| from lineitem limit 100""".stripMargin) {
checkGlutenPlan[ProjectExecTransformer]
}
// Multiple calls must produce independent results (issue #7628).
runQueryAndCompare(
"""SELECT monotonically_increasing_id(), monotonically_increasing_id()
| from lineitem limit 100""".stripMargin) {
checkGlutenPlan[ProjectExecTransformer]
}
}

test("sequence function optimized by Spark constant folding") {
Expand Down
8 changes: 8 additions & 0 deletions cpp/velox/compute/WholeStageResultIterator.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -666,6 +666,14 @@ std::unordered_map<std::string, std::string> WholeStageResultIterator::getQueryC

configs[velox::core::QueryConfig::kSparkPartitionId] = std::to_string(taskInfo_.partitionId);

// Disable expression deduplication for non-deterministic functions to
// match Spark semantics. Spark creates separate instances for each
// non-deterministic expression, each with independent state. Without
// this, Velox merges structurally identical non-deterministic calls
// (e.g. two monotonically_increasing_id() in the same query) into one
// shared instance, causing incorrect results.
configs[velox::core::QueryConfig::kExprDedupNonDeterministic] = "false";

// Enable Spark legacy date formatter if spark.sql.legacy.timeParserPolicy is set to 'LEGACY'
// or 'legacy'
if (veloxCfg_->get<std::string>(kSparkLegacyTimeParserPolicy, "") == "LEGACY") {
Expand Down
1 change: 1 addition & 0 deletions gluten-substrait/src/main/scala/org/apache/gluten/expression/ExpressionMappings.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,7 @@ object ExpressionMappings {
Sig[MakeDecimal](MAKE_DECIMAL),
Sig[PromotePrecision](PROMOTE_PRECISION),
Sig[SparkPartitionID](SPARK_PARTITION_ID),
Sig[MonotonicallyIncreasingID](MONOTONICALLY_INCREASING_ID),
Sig[AtLeastNNonNulls](AT_LEAST_N_NON_NULLS),
Sig[WidthBucket](WIDTH_BUCKET),
Sig[ReplicateRows](REPLICATE_ROWS),
Expand Down