-
Notifications
You must be signed in to change notification settings - Fork 582
Description
Description
Some Spark 3.2-specific compatibility code remains in the common codebase:
#gluten-core/src/main/scala/org/apache/spark/util/SparkVersionUtil.scala
Defines lteSpark32 (used by other files)
#gluten-core/src/main/scala/org/apache/spark/util/SparkTaskUtil.scala
Lines 67-82: Spark 3.2-specific constructor path for TaskContext
#gluten-core/src/main/scala/org/apache/spark/util/SparkPlanUtil.scala
Lines 26-28: Spark 3.2-specific path for supportsRowBased
#gluten-core/src/main/scala/org/apache/spark/sql/execution/adaptive/GlutenCostEvaluator.scala
Lines 33-35: Spark 3.2-specific CostEvaluator instantiation
#gluten-core/src/main/scala/org/apache/gluten/extension/columnar/transition/Convention.scala
Lines 174-184: Spark 3.2-specific row type handling
#backends-velox/src/test/scala/org/apache/gluten/execution/MiscOperatorSuite.scala
Lines 911-919: Test that checks for Spark 3.2
These items are compatibility shims in shared code that can be cleaned up separately.
Also, need to clean up the shims/spark32 directory with:
shims/spark32/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnarArrayShim.java
shims/spark32/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFooterReaderShim.scala
Related Issue: #8960
Gluten version
None