Running Apache Gluten locally on macOS (Apple Silicon) with Spark 3.5 – libgluten.dylib missing

[soumilshah1995]

Hi Gluten community,
I’m trying to run Apache Gluten locally on macOS (Apple Silicon / M1) with Apache Spark 3.5.3, and I’m running into a native library loading issue.
I wanted to check whether local macOS support is expected to work today, or if Gluten is currently Linux-only for the Velox backend.

OS: macOS (Apple Silicon – aarch64)
Spark: 3.5.3 (spark-3.5.3-bin-hadoop3)
Java: OpenJDK 17
Gluten:
gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.5.0.jar
Backend: Velox
Execution mode: local

$SPARK_HOME/bin/spark-shell \
  --jars $GLUTEN_JAR \
  --conf spark.plugins=org.apache.gluten.GlutenPlugin \
  --conf spark.gluten.sql.enable=true \
  --conf spark.gluten.execution.backend=velox \
  --conf spark.memory.offHeap.enabled=true \
  --conf spark.memory.offHeap.size=2g \
  --conf spark.driver.extraClassPath=$GLUTEN_JAR \
  --conf spark.executor.extraClassPath=$GLUTEN_JAR \
  --conf spark.sql.columnVector.offheap.enabled=true \
  --conf spark.gluten.sql.fallback.enabled=true \
  --conf spark.driver.bindAddress=127.0.0.1 \
  --conf spark.driver.host=127.0.0.1

Error Observed

WARN Discovery: Found duplicated component class
org.apache.gluten.component.VeloxPaimonComponent in the classpath, ignoring.

ERROR SparkContext: Error initializing SparkContext.
org.apache.gluten.exception.GlutenException:
java.io.FileNotFoundException: darwin/aarch64/libgluten.dylib
    at org.apache.gluten.jni.JniLibLoader.load(JniLibLoader.java:89)

Questions
Is running Gluten + Velox locally on macOS supported today?
Are there prebuilt macOS (darwin/aarch64) native binaries for Gluten?

I took jar from here
https://downloads-he-de-2.apache.org/incubator/gluten/1.4.0-incubating/apache-gluten-1.4.0-incubating-bin-spark35.tar.gz

[jinchengchenghh]

Please see the document #11547