[Spark 4.1.0] StoragePartitionJoinParams moved to different package

[res-life]

Description

In Spark 4.1.0, StoragePartitionJoinParams was moved to a different package.

Spark 4.0.x

package org.apache.spark.sql.execution.datasources.v2
// Defined inside BatchScanExec.scala
case class StoragePartitionJoinParams(...)

Spark 4.1.0

package org.apache.spark.sql.execution.joins
// Moved to separate file: StoragePartitionJoinParams.scala
case class StoragePartitionJoinParams(...)

Impact

Code that imports StoragePartitionJoinParams from the old package will fail to compile:

object StoragePartitionJoinParams is not a member of package org.apache.spark.sql.execution.datasources.v2

Affected Files

• sql-plugin/src/main/spark350db143/scala/com/nvidia/spark/rapids/shims/GpuBatchScanExec.scala

Solution

Create a shim StoragePartitionJoinShims that provides a type alias:

For Spark 3.5.0-db143 / 4.0.x (spark350db143/):

import org.apache.spark.sql.execution.datasources.v2.StoragePartitionJoinParams

object StoragePartitionJoinShims {
  type SpjParams = StoragePartitionJoinParams
  def default(): SpjParams = StoragePartitionJoinParams()
}

For Spark 4.1.0+ (spark410/):

import org.apache.spark.sql.execution.joins.StoragePartitionJoinParams

object StoragePartitionJoinShims {
  type SpjParams = StoragePartitionJoinParams
  def default(): SpjParams = StoragePartitionJoinParams()
}

Update consuming code to use the shim:

// Before
import org.apache.spark.sql.execution.datasources.v2.StoragePartitionJoinParams
spjParams: StoragePartitionJoinParams = StoragePartitionJoinParams()

// After
import com.nvidia.spark.rapids.shims.StoragePartitionJoinShims
spjParams: StoragePartitionJoinShims.SpjParams = StoragePartitionJoinShims.default()

Note

The case class definition itself is identical between versions - only the package location changed.

References

• Spark 4.0.1: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/BatchScanExec.scala
• Spark 4.1.0: sql/core/src/main/scala/org/apache/spark/sql/execution/joins/StoragePartitionJoinParams.scala

[res-life]

I'll complete the shim, but need to double check if have other impact.

The case class definition itself is identical between versions
Seems no impact, but better to do a double check, e.g.: run related IT cases.

[res-life]

I'll finish the shim and make Spark 410 shim building pass, but need more investigations about potential impacts, tests.

[res-life]

Will be closed by #14120