Update resource configurations and add volume mounts in Argo workflow templates

[Dashing-Nelson]

Working Argo Workflow

https://argo.platform.prod.everycure.org/workflows/argo-workflows/spoke-xgb-ensemble-b6df9a87?uid=1fead150-e3e4-4897-b3bf-7aabd477dd82

Description of the changes

This pull request introduces important updates to the Argo workflow templates and pipeline resource configurations, focusing on improved resource allocation and ephemeral storage management for Neo4j and related nodes. The main themes are enhanced resource requests/limits and the addition of an ephemeral scratch volume for Neo4j data directories.

Resource configuration updates:

• Increased memory requests and limits for the filtering pipeline node from 75 to 256 units, ensuring more resources are available for filtering tasks.
• Set explicit CPU (48 units) and memory (350 units) requests/limits, as well as ephemeral storage (1024 units), for the topological embeddings pipeline node to better control resource usage.

Ephemeral storage and Neo4j configuration:

• Added a scratch ephemeral volume to the Argo workflow spec, mounted at /data, and configured Neo4j to use this volume for all its data, logs, import, and metrics directories. This ensures that Neo4j has isolated, high-performance temporary storage during workflow execution. [1] [2] [3] [4]

Submodule update:

• Updated the infra/secrets submodule to a new commit, likely pulling in the latest secret management changes.

Fixes / Resolves the following issues:

• OOM Error for Topological embeddings

Checklist:

• Added label to PR (e.g. enhancement or bug)
• Ensured the PR is named descriptively. FYI: This name is used as part of our changelog & release notes.
• Looked at the diff on github to make sure no unwanted files have been committed.