Skip to content

Remove Spark 3.2 compatibility code#11731

Draft
QCLyu wants to merge 5 commits intoapache:mainfrom
QCLyu:qclyu
Draft

Remove Spark 3.2 compatibility code#11731
QCLyu wants to merge 5 commits intoapache:mainfrom
QCLyu:qclyu

Conversation

@QCLyu
Copy link
Contributor

@QCLyu QCLyu commented Mar 10, 2026

Summary

  • Remove all lteSpark32 branches from shared code (dead code since Spark 3.2 is no longer supported)
  • Clean up SparkVersionUtil.scala (remove lteSpark32, lteSpark33)
  • Delete orphaned shims/spark32/ files and Spark 3.2-specific test branches
  • Rename KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault

Test plan

  • Build with Spark 3.3, 3.4, 3.5 profiles to verify no compilation errors
  • Run existing unit tests (no behavioral changes, only dead code removal)

Partially assisted by Claude Opus 4.6.

Closes #11379

@github-actions github-actions bot added CORE works for Gluten Core VELOX labels Mar 10, 2026
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@QCLyu QCLyu changed the title Qclyu Remove Spark 3.2 compatibility code Mar 10, 2026
@QCLyu
Copy link
Contributor Author

QCLyu commented Mar 10, 2026

Hi @PHILO-HE , just thought carefully about your comment in the previous PR: #11495 (comment) This time I renamed KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault, instead of completely merging it with KnownRowType.

My rationale is "this: SparkPlan =>" created a self-type constraint: If you put this on KnownRowType directly, then any implementor of KnownRowType would have to be a SparkPlan. 

Please correct me if I'm wrong.

@QCLyu QCLyu marked this pull request as draft March 10, 2026 05:25
@github-actions
Copy link

Run Gluten Clickhouse CI on x86

@PHILO-HE
Copy link
Member

Hi @PHILO-HE , just thought carefully about your comment in the previous PR: #11495 (comment) This time I renamed KnownRowTypeForSpark33OrLater to KnownRowTypeWithDefault, instead of completely merging it with KnownRowType.

My rationale is "this: SparkPlan =>" created a self-type constraint: If you put this on KnownRowType directly, then any implementor of KnownRowType would have to be a SparkPlan. 

Please correct me if I'm wrong.

@QCLyu, the constraint is acceptable maybe. This trait was introduced to categorize Spark Plans based on the data format. So I would advise merging them. Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remove remaining Spark 3.2-specific compatibility code

2 participants