WIP: Add support for running with a remote Armada server by richscott · Pull Request #83 · armadaproject/armada-spark

richscott · 2025-11-21T23:27:48Z

These changes add support for running a Spark client against a remote Armada server (as opposed to both being on localhost). This has been developed and tested on a macOS system as the Spark client with an Ubuntu Linux server running Armada (on top of a kind k8s cluster).

See updates to the README.md for the steps to use this.

Change Details:

support for using TLS certificates is added - instructions have been added to README.md detailing on how to get the cert files, define ARMADA_MASTER in scripts/config.sh, etc.
the dev-e2e.sh has been modified to check for the existence and health of the specified Armada server, but if ARMADA_MASTER is not localhost, it will not attempt to start the server via armada-operator.
the dev-e2e.sh script now does not use Docker to run the tests - it directly uses the local clone-copy of Spark (which is typically in .spark-3.5.5).
the code to acquire a fabric8 k8s client has been changed from the default client (which assumes its running inside the destination cluster), to build the client, and use the TLS cert files if specified in the init.sh/config.sh configuration settings.

Changes to have submitArmadaSpark.sh work with a possibly-remote Armada server, just as the dev-e2e.sh logic, are not yet included in this PR, but will be coming soon.

These changes are to allow running an armada-spark client against a remote (i.e. non-localhost) Armada cluster server. Signed-off-by: Rich Scott <richscott@sent.com>

Also, fixes in TestOrchestrator for running against a remote Armada instance, and run the tests directly, instead of using a Docker container on the client. Signed-off-by: Rich Scott <richscott@sent.com>

It will soon be used by ArmadaClientApplication. Signed-off-by: Rich Scott <richscott@sent.com>

Signed-off-by: Rich Scott <richscott@sent.com>

…artup tasks Signed-off-by: Rich Scott <richscott@sent.com>

…rver. Signed-off-by: Rich Scott <richscott@sent.com>

GeorgeJahad · 2025-11-22T00:40:08Z

README.md

+of `kubectl config view`. These files can be left in this directory.
+


Should e2e/*crt be in .gitignore?

GeorgeJahad · 2025-11-22T00:45:25Z

scripts/dev-e2e.sh


-  TMPDIR="$scripts/.tmp" "$AOHOME/bin/tooling/kind" load docker-image "$IMAGE_NAME" --name armada 2>&1 \
-   | log_group "Loading Docker image $IMAGE_NAME into Armada cluster";
+  if [ "$ARMADA_MASTER" = "localhost" ] ; then


ARMADA_MASTER is never just "localhost" is it?

usually something like: "armada://localhost:30002"

GeorgeJahad · 2025-11-22T01:00:53Z

src/test/scala/org/apache/spark/deploy/armada/e2e/ArmadaOperations.scala

-    Seq(armadactlCmd) ++ subCommand.split(" ") ++ Seq("--armadaUrl", armadaUrl)
+    // armadactl command expects the server address to be of the form
+    // <hostname-or-IP>:<port> with no pseudo-protocol prefix
+    var armadactlUrl = armadaUrl.replaceFirst("^armada://", "")


GeorgeJahad · 2025-11-22T01:04:57Z

src/test/scala/org/apache/spark/deploy/armada/e2e/TestAssertions.scala

 package org.apache.spark.deploy.armada.e2e

 import io.fabric8.kubernetes.api.model.Pod
+import org.apache.spark.deploy.armada.K8sClient


is this used?

GeorgeJahad · 2025-11-22T01:07:17Z

src/test/scala/org/apache/spark/deploy/armada/e2e/TestOrchestrator.scala

-    val dockerCommand = buildDockerCommand(
+    val runTestCommand = buildRunTestCommand(
      config.imageName,
-      volumeMounts,


why was this removed?

GeorgeJahad · 2025-11-22T01:08:45Z

src/test/scala/org/apache/spark/deploy/armada/e2e/TestOrchestrator.scala

      lookoutUrl: String,
      pythonScript: Option[String]
  ): Seq[String] = {
+    val sparkRepoCopy = ".spark-3.5.5"


GeorgeJahad · 2025-11-22T01:08:56Z

src/test/scala/org/apache/spark/deploy/armada/e2e/TestOrchestrator.scala

+
+    val classPathEntries: Seq[String] = Seq(
+      ".",
+      s"${sparkRepoCopy}/assembly/target/scala-2.13/jars/*",


hard coded versions?

GeorgeJahad · 2025-11-22T01:18:00Z

Running with:

ARMADA_MASTER=armada://localhost:30002

no longer seems to work:

scripts/submitArmadaSpark.sh 100
Submitting spark job to Armada.
/home/gbj/incoming/armada-spark/scripts/init.sh: line 67: CLIENT_CERT_FILE: unbound variable

Signed-off-by: Rich Scott <richscott@sent.com>

Get the first external interface IP address and use in Kind configuration for allowing external K8S/Armada API access. Add protective quotes around TLS vars in dev-e2e.sh, per shellcheck. Use `realpath` for getting reliable full pathnames. Signed-off-by: Rich Scott <richscott@sent.com>

Use better pattern checks for verifying if Armada master is localhost; remove quote wrapper around tls_args, so Maven doesn't error on a "" target. Signed-off-by: Rich Scott <richscott@sent.com>

Signed-off-by: Rich Scott <richscott@sent.com>

Also, derive K8S server address (especially the port, as `kind` dynamically selects a random port number to expose the api-server) from ~/.kube/config. Signed-off-by: Rich Scott <richscott@sent.com>

richscott added 9 commits November 17, 2025 16:56

Start adding support for running against remote Armada cluster server.

b635db5

These changes are to allow running an armada-spark client against a remote (i.e. non-localhost) Armada cluster server. Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

49aa0a7

Add script for getting TLS cert files for remote server

b14a8e8

Also, fixes in TestOrchestrator for running against a remote Armada instance, and run the tests directly, instead of using a Docker container on the client. Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

16fc24e

Move K8sClient class from src/test into src/main

7d3a456

It will soon be used by ArmadaClientApplication. Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

5bc5858

Remove debugging messages; re-enable init-cluster on e2e script.

aa1e144

Signed-off-by: Rich Scott <richscott@sent.com>

In E2E script, if Armada server is not localhost, don't do certain st…

c538b1b

…artup tasks Signed-off-by: Rich Scott <richscott@sent.com>

Add more README.md content on setting up for using a remote Armada se…

0c06f07

…rver. Signed-off-by: Rich Scott <richscott@sent.com>

richscott requested review from GeorgeJahad and dejanzele November 21, 2025 23:27

richscott self-assigned this Nov 21, 2025

GeorgeJahad reviewed Nov 22, 2025

View reviewed changes

richscott added 10 commits November 24, 2025 08:58

Quote evaluated script vars, for CI checker

fee4552

Signed-off-by: Rich Scott <richscott@sent.com>

Disable building/testing for Spark 3.5.5 config for now.

e4e4a83

Signed-off-by: Rich Scott <richscott@sent.com>

Scala linter fixes; Bash lint fixes

7e49435

Signed-off-by: Rich Scott <richscott@sent.com>

Add ARMADA_LOOKOUT_URL to init.sh

2b705ae

Signed-off-by: Rich Scott <richscott@sent.com>

Check if TLS cert vars are defined before referencing

cc7639a

Signed-off-by: Rich Scott <richscott@sent.com>

Conditionally add TLS cert properties to test invocation

b3d4265

Signed-off-by: Rich Scott <richscott@sent.com>

Check for busybox image for init container

8127bcb

Use better pattern checks for verifying if Armada master is localhost; remove quote wrapper around tls_args, so Maven doesn't error on a "" target. Signed-off-by: Rich Scott <richscott@sent.com>

Add init container image name.

5d51466

Signed-off-by: Rich Scott <richscott@sent.com>

Add diagnostic search log for .spark-* directory

1172550

Signed-off-by: Rich Scott <richscott@sent.com>

richscott added 11 commits December 2, 2025 14:04

More GH Actions debugging

a0c0d8a

Signed-off-by: Rich Scott <richscott@sent.com>

More diagnostics for finding spark source dir

233e413

Signed-off-by: Rich Scott <richscott@sent.com>

Clone the Spark repo to get bin/spark-class and jars.

9e0b309

Signed-off-by: Rich Scott <richscott@sent.com>

Always rebuild the Spark jars before running E2E.

6345155

Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

4a8b70d

Signed-off-by: Rich Scott <richscott@sent.com>

Fix Scala formatting violations.

3e26371

Signed-off-by: Rich Scott <richscott@sent.com>

Remove more refs and logic for old local:// prefix.

2347697

Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

2d71d2f

Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

332b20c

Signed-off-by: Rich Scott <richscott@sent.com>

Merge branch 'master' into rich/support-remote-server

c4f9091

Return to running client code in Docker image.

8f94b07

Also, derive K8S server address (especially the port, as `kind` dynamically selects a random port number to expose the api-server) from ~/.kube/config. Signed-off-by: Rich Scott <richscott@sent.com>

richscott marked this pull request as draft March 30, 2026 01:20

richscott requested a review from sudiptob2 March 30, 2026 01:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add support for running with a remote Armada server#83

WIP: Add support for running with a remote Armada server#83
richscott wants to merge 30 commits intomasterfrom
rich/support-remote-server

richscott commented Nov 21, 2025 •

edited

Loading

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad Nov 22, 2025

Uh oh!

GeorgeJahad commented Nov 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		of `kubectl config view`. These files can be left in this directory.

Conversation

richscott commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

GeorgeJahad commented Nov 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

richscott commented Nov 21, 2025 •

edited

Loading