add history server and "--files" support

[GeorgeJahad]

What: Add --files support for Armada cluster-mode submissions and enable automatic S3-backed event logging for Spark History Server.

Why: Armada's cluster-mode driver needs to resolve locally-submitted files (via --files) to their uploaded remote paths so executors can access them. Without this, file distribution silently fails. Event logging to S3 enables post-hoc job inspection via Spark History Server.

Changes:

• Add resolveLocalAppResource to map local app resources to their feature-step-uploaded remote URIs (e.g., S3) using the --class index position in container args
• Add applyFileUploadProperties to propagate resolved spark.jars, spark.files, spark.archives, spark.submit.pyFiles from driver feature steps into SparkConf
• Set spark.kubernetes.submitInDriver=true so the driver pod downloads remote files locally on startup
• Add isArmadaClusterModeDriver and | ARMADA to OptionAssigner entries for files/archives/jars in all three version-specific SparkSubmit.scala files (3.3, 3.5, 4.1)
• Add S3_CONF and EVENT_LOG_CONF arrays to init.sh for automatic S3 credential and event log configuration
• Add scripts/runHistoryServer.sh to launch a Spark History Server reading event logs from S3, with error detection if the log directory doesn't exist
• Add scripts/filesParameterExample.sh demonstrating --files parameter with UUID-tagged CSV distributed to executors
• Add EVENT_LOG_CONF and S3_CONF to submitArmadaSpark.sh so jobs automatically capture event logs when S3 is configured

Tests:

• 6 new resolveLocalAppResource tests covering local/remote paths, S3 resolution, missing --class, and missing container
• 1 new test verifying driverSystemProperties from feature steps are populated in ArmadaJobConfig

How to verify:

1. mvn test -DwildcardSuites=org.apache.spark.deploy.armada.submit.ArmadaClientApplicationSuite — all tests pass
2. Run scripts/filesParameterExample.sh — executors print UUID from distributed CSV file
3. Run scripts/submitArmadaSpark.sh -p 100 — Pi job completes, event logs written to S3
4. Run scripts/runHistoryServer.sh — History Server UI at http://localhost:18080 shows completed jobs

[Copilot]

Pull request overview

Adds first-class support for Spark’s distributed resource flags (--files, --jars, --archives) when submitting to Armada (including “submit-in-driver” behavior), and introduces scripts to enable S3-backed Spark event logging plus running a Spark History Server against those logs.

Changes:

• Propagate file-upload-related system properties from Spark K8s feature steps into the SparkConf used for Armada driver/executor submission.
• Extend SparkSubmit option assignment and archive-download behavior to include the ARMADA cluster manager for multiple Spark versions (3.3/3.5/4.1).
• Add/init scripts to configure S3 credentials + Spark event logging and provide History Server + --files usage examples.

Reviewed changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
src/main/scala/org/apache/spark/deploy/armada/submit/ArmadaClientApplication.scala	Adds driver feature-step system properties propagation and local app resource resolution logic.
src/main/scala-spark-3.3/org/apache/spark/deploy/SparkSubmit.scala	Enables ARMADA for --files/--archives/--jars and driver-side resource download behavior.
src/main/scala-spark-3.5/org/apache/spark/deploy/SparkSubmit.scala	Same as above for Spark 3.5.
src/main/scala-spark-4.1/org/apache/spark/deploy/SparkSubmit.scala	Same as above for Spark 4.1.
src/test/scala/org/apache/spark/deploy/armada/submit/ArmadaClientApplicationSuite.scala	Adds tests for new resource-resolution behavior and updates fixtures for new config field.
src/test/scala/org/apache/spark/deploy/armada/e2e/ArmadaSparkE2E.scala	Adjusts python e2e spark conf (removes upload-path setting).
scripts/init.sh	Adds S3_CONF and EVENT_LOG_CONF helpers for S3 credentials + event logging.
scripts/submitArmadaSpark.sh	Injects S3/event-log config into standard Armada submit script.
scripts/runHistoryServer.sh	New script to run Spark History Server against S3 event logs with basic error detection.
scripts/filesParameterExample.sh	New example demonstrating --files distribution to executors.
.gitignore	Ignores generated example/files directory from the example script.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[EnricoMi]

Couldn't you say everything that is ! isLocalFile is remote?

[GeorgeJahad]

thanks I simplified this

[sudiptob2]

Looks good, a few minor comments.

[sudiptob2]

LGTM

[GeorgeJahad]

thanks for the reviews @sudiptob2 and @EnricoMi