Deployment Guide for Spark on EMR

[ammarchalifah]

Hi, sorry for the amateur question, but I'm facing difficulty when testing the deployment of Gluten+Velox on AWS EMR.

I'm using EMR 7.8, Spark 3.5.4, with PySpark. I'm using Intel x86 instances on Amazon Linux distribution. I want to test a job that reads from vanilla Parquet, do simple Project & Exchange, and write to Iceberg table (copy-on-write). Both input & output are in S3.

I downloaded the gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar and put it into S3 directly. Then, in the EMR cluster, I'm trying to run a Spark submit while passing the JAR and setting up the configurations.

Questions:

• Does this approach works in general, or should I build from source in this case?
• Is there any community guide regarding building/deploying Gluten-Velox on EMR that is generic & easy to follow?

[FelixYBW]

Gluten uses couple of Spark APIs which may be modified by customized Spark like AWS EMR. Gluten is built on top of vanilla spark only. So you will have high risk to encounter API conflict when you run Gluten with EMR spark.

[FelixYBW]

The only way is to use vanilla spark + Gluten without EMR.

Or ping AWS to see if they provide native engine acceleration.

[ammarchalifah]

Got it. I'm currently just doing testing and want to see how far Gluten on EMR goes, and if it breaks, at which point it breaks. Purely for experiment.

The way I'm doing the deployment right now is

• Downloading the JAR gluten-velox-bundle-spark3.5_2.12-linux_amd64-1.4.0.jar and dump it to S3
• During Spark submit, pass extra parameters:

--conf spark.yarn.dist.files={s3_jar_path}
--conf spark.plugins=org.apache.gluten.GlutenPlugin
--conf spark.memory.offHeap.enabled=true
--conf spark.memory.offHeap.size=20g
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager
--conf spark.driver.extraClassPath={jar_path}
--conf spark.executor.extraClassPath={jar_path}

The Spark application is properly booted up with the plugin

[ammarchalifah]

But I'm facing difficulty with reading Parquet file from S3. Without Gluten, reading Parquet from S3 with

return (
            spark.read.schema(input_data_schema)
            .option("basePath", base_path)
            .option("recursiveFileLookup", "true")
            .parquet(*paths)
        )

works fine. With Gluten enabled, it just behaves as if the file doesn't exist and returned an empty dataframe.

[FelixYBW]

I don't think anyone here have such experience :(

[ammarchalifah]

Does the distribution already contains the S3 related modules? I realized I haven't explicitly set these up, maybe it's related

https://github.com/apache/incubator-gluten/blob/main/docs/get-started/VeloxS3.md

[ammarchalifah]

Small update, managed to start the cluster with the JAR stored from s3, but then unable to read the Parquet files from S3. EMR has preinstalled packages that manages EMRFS (allowing EMR to access s3://) that doesn't work with Gluten. Then, tried installing extra packages org.apache.hadoop:hadoop-aws:3.3.4,com.amazonaws:aws-java-sdk-bundle:1.12.262 to manage the connection with S3 and use s3a:// instead of s3://.

Didn't really manage to get it working and gave up, maybe I'll come back to this at some point.

If anyone managed to run Gluten-Velox on EMR, looking forward to follow your guidance :)