[GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic#10823
[GLUTEN-10648][VL] Support Iceberg overwrite partitions dynamic#10823jinchengchenghh merged 2 commits intoapache:mainfrom
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
It is public, username gluten password hN2xX3uQ4m |
|
This is an unstable test, I will create a PR to fix it. |
|
The line number is different with source ClickHouseAdaptiveQueryExecSuite.scala:84 |
| override def supportOverwriteByExpression(): Boolean = | ||
| GlutenConfig.get.enableOverwriteByExpression && enableEnhancedFeatures() | ||
|
|
||
| override def supportOverwritePartitionsDynamic(): Boolean = |
There was a problem hiding this comment.
Here is the backend setting config, just return enableEnhancedFeatures(), and this change can trigger the CI again.
|
Run Gluten Clickhouse CI on x86 |
|
|
||
| def supportOverwriteByExpression(): Boolean = false | ||
|
|
||
| def supportOverwritePartitionsDynamic(): Boolean = false |
There was a problem hiding this comment.
Unnecessarily related to this PR, but we do need to have the new V2 columnar write operators covered in tests individually without having to enable the Iceberg writer, as they were design to be general. Vanilla Spark uses an in-memory catalog for testing the row-based V2 write operators. We may want to introduce something similar just for testing. #9896
There was a problem hiding this comment.
Sounds good, v2 write is a general capability that can be used in all other lake formats. I know a bit about DSv2, and happy to help if needed.
There was a problem hiding this comment.
Thanks! Feel free to open issues and PRs.
There was a problem hiding this comment.
Spark uses an in-memory catalog for testing the row-based V2 write operators. We may want to introduce something similar just for testing.
@Zouxxyy Just recalled that other contributors might already work on something similar, let me confirm first to avoid duplicated work. :) I don't have their GitHub ID or Email at this moment but I will try to get them into the public discussion.
There was a problem hiding this comment.
I've confirmed, they are not working on the test topic. So feel free to take if wanted. We may have public discussions about the further matters later on.
There was a problem hiding this comment.
Thanks, I'd like to, testing is the foundation, which will make the integration more directional and reliable.
| .createWithDefault(true) | ||
|
|
||
| val COLUMNAR_OVERWRIET_PARTITIONS_DYNAMIC_ENABLED = | ||
| buildConf("spark.gluten.sql.columnar.overwriteOverwritePartitionsDynamic") |
There was a problem hiding this comment.
Hi @Zouxxyy,
Should this be spark.gluten.sql.columnar.overwritePartitionsDynamic?
There was a problem hiding this comment.
Sorry, my mistake, I might have copied the wrong content.
This should fix CI error on `AllVeloxConfiguration`.
This should fix CI error on `AllVeloxConfiguration`.
3 participants
What changes are proposed in this pull request?
Support Iceberg overwrite partitions dynamic
How was this patch tested?