Facing issue with custom build with enable_vcpkg=OFF

[himanshu-zetta]

I am building gluten with velox backend and enable_vcpkg=OFF following this:

With config enable_vcpkg=OFF, not all dependency libraries will be statically linked, instead the script will install the libraries to system then pack the dependency libraries into another jar named gluten-package-${Maven-artifact-version}.jar. Then you need to add the jar to extraClassPath and set spark.gluten.loadLibFromJar=true. Otherwise, you need to install shared dependency libraries on each worker node. You may find the libraries list from the gluten-package jar.

But when I'm running with compile code with above configuration for classPath and loadLibFromJar, getting this error:

java.lang.NoClassDefFoundError: Could not initialize class org.apache.gluten.GlutenPlugin$
  at org.apache.gluten.GlutenDriverPlugin.setPredefinedConfigs(GlutenPlugin.scala:139)
  at org.apache.gluten.GlutenDriverPlugin.init(GlutenPlugin.scala:69)
  at org.apache.spark.internal.plugin.DriverPluginContainer.$anonfun$driverPlugins$1(PluginContainer.scala:53)
  at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:293)
  at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
  at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)

Please let me know how to fix this issue?

[PHILO-HE]

@himanshu-zetta, could you try to use vcpkg=ON? It makes all third-party libs statically linked. The dynamic link path is going to be deprecated.