Skip to content

Remove remaining Spark 3.2-specific compatibility code #11379

@QCLyu

Description

@QCLyu

Description

Some Spark 3.2-specific compatibility code remains in the common codebase:
#gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala
Defines lteSpark32 (used by other files)
#gluten-core/src/main/scala/org/apache/spark/util/SparkTaskUtil.scala
Lines 67-82: Spark 3.2-specific constructor path for TaskContext
#gluten-core/src/main/scala/org/apache/spark/util/SparkPlanUtil.scala
Lines 26-28: Spark 3.2-specific path for supportsRowBased
#gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala
Lines 33-35: Spark 3.2-specific CostEvaluator instantiation
#gluten-core/src/main/scala/org/apache/gluten/extension/columnar/transition/Convention.scala
Lines 174-184: Spark 3.2-specific row type handling
#backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
Lines 911-919: Test that checks for Spark 3.2

These items are compatibility shims in shared code that can be cleaned up separately.

Also, need to clean up the shims/spark32 directory with:
shims/spark32/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarArrayShim.java
shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFooterReaderShim.scala
Related Issue: #8960

Gluten version

None

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions