question job-json-overides vs launcher.templates

[airpaio]

On Workbench, we can set job-json-overides with different profiles which allows for some dynamicism between different user/groups. For instance we might have the below profiles for groups which use different service accounts and node afinities defined in the job-json-overides.

I really have 2 questions here:

1. How can we get the same dynamics for specifying different service accounts/node afinities using the launcher templates?
2. Below we have two profiles for users in group2 (one with job-json-overides specifying node afinity for gpu nodes and the other does not). Assuming the user is in both group2 and group2_gpu, the Session parameters would allow the user to specify number of GPUs. What happens if a user launches a session with 0 GPUs specified? Would the session launch on the GPU nodes or not? Would the session perhaps launch onto the GPU nodes, but not use any GPUs? Is there any other way to allow for the dynamic selection of node groups when a user launches a session?

"@group1":
  default-cpus: 4
  default-mem-mb: 2048
  default-nvidia-gpus: 0
  allow-unknown-images: 1
  job-json-overrides:  # use group1 service account and afinity for small nodes
    - name: service-account
      target: /spec/template/spec/group1ServiceAccount
      json: "group1-service-account"
    - name: node-affinity
      target: /spec/template/spec/affinity
      json:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: nodegroup-label
                  operator: "In"
                  values:
                    - small
"@group2":
  default-cpus: 4
  default-mem-mb: 4096
  default-nvidia-gpus: 0
  max-cpus: 8
  allow-unknown-images: 1
  job-json-overrides:  # use group2 service account and afinity for large nodes
    - name: service-account
      target: /spec/template/spec/group2ServiceAccount
      json: "group2-service-account"
    - name: node-affinity
      target: /spec/template/spec/affinity
      json:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: nodegroup-label
                  operator: "In"
                  values:
                    - large
"@group2_gpu":  # group2 with access to gpu nodes
  default-cpus: 4
  default-mem-mb: 4096
  default-nvidia-gpus: 2
  max-cpus: 8
  allow-unknown-images: 1
  job-json-overrides:  # use group2 service account and afinity for gpu nodes
    - name: service-account
      target: /spec/template/spec/group2ServiceAccount
      json: "group2-service-account"
    - name: node-affinity
      target: /spec/template/spec/affinity
      json:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                - key: nodegroup-label
                  operator: "In"
                  values:
                    - gpu

[colearendt]

Howdy @airpaio !

These are great questions - thanks for the thorough writeup!

You are right that templates are (today, in the helm charts) not quite so flexible at changing things per group. However, I think you may be able to get by with a few settings:

• For node selection, you can set up placement-constraints, which get injected into jobs as nodeSelectors, and show up in the UI for users to select
• For GPU selection, you can use the GPU selection parameters
• These are both set per group in launcher.kubernetes.profiles.conf and do not involve templates
• Moreover, it allows users to choose when they want to run on particular node groups or GPUs, without requiring that every job they run behaves the same way.

I hope that helps! Basically the idea is to drop the use of job-json-overrides entirely 😄

https://docs.posit.co/job-launcher/kube.html#kube-config

[MariaSemple]

Hey @airpaio, I hope Cole's response helped you configure your environment the way you wanted! Since it looks like this question is answered now, I'm going to close this issue, but if you have further questions, please feel free to comment/re-open.