[GLUTEN-11683][VL] Enable GlutenParquetTypeWideningSuite for Spark 4.0 and 4.1#11684
Merged
baibaichen merged 2 commits intoapache:mainfrom Mar 8, 2026
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
bb4c852 to
ed359a2
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
ed359a2 to
ba6f68b
Compare
|
Run Gluten Clickhouse CI on x86 |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
ba6f68b to
115b01f
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
115b01f to
89628ce
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
89628ce to
556c23a
Compare
Add translateException() to ClosableIterator as a virtual hook for backend-specific exception translation. Override in ColumnarBatchOutIterator to translate Velox type conversion errors into Spark's SchemaColumnConvertNotSupportedException. Exclude 63 tests: Velox native reader always rejects incompatible type conversions (no parquet-mr fallback), and does not support DELTA_BYTE_ARRAY encoding for FIXED_LEN_BYTE_ARRAY decimals. Test results: 21 pass / 63 ignored (spark40 and spark41). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
baibaichen
force-pushed
the
pr1/enable-parquet-type-widening-suite
branch
from
556c23a to
4f0f7a9
Compare
|
Run Gluten Clickhouse CI on x86 |
1 similar comment
|
Run Gluten Clickhouse CI on x86 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Enable
GlutenParquetTypeWideningSuite(84 tests) for Spark 4.0 and 4.1, validating Parquet type widening support (SPARK-40876).Changes
Exception translation (
ClosableIterator.java+ColumnarBatchOutIterator.java):Add
translateException()virtual hook to convert Velox type conversion errors toSchemaColumnConvertNotSupportedException.Enable suite (
VeloxTestSettings.scalafor spark40/spark41):Enable suite with 63 excludes for tests not yet supported by Velox.
Build improvement (
builddep-veloxbe-inc.sh):Target specific CMake build targets for faster incremental builds.
Test Results
The 63 excluded tests require Velox C++ type widening support and will be addressed in follow-up PRs.
Fixes #11683
How was this patch tested?
Ran
GlutenParquetTypeWideningSuitelocally for both Spark 4.0 and 4.1: 21 pass / 0 fail / 63 ignored.Was this patch authored or co-authored using generative AI tooling?
Yes, co-authored with GitHub Copilot.