allow-custom-resources and workbench jobs

[murraj]

Are there any known issues with not allowing allow-custom-resources (allow-custom-resources=0) and workbench jobs? If I turn it on, the UI seems to respect it. However user cannot kick off workbench job.

We are running workbench in AKS. We are using the same cluster to dispatch workbench jobs. I noticed that if I attempt to set allow-custom-resources: 0 in the helm chart, we get the "Cannot start job" error when attempting to kick off a workbench job. However, if I remove allow-custom-resources (or set allow-custom-resources: 1), it works properly.

Ideally, I would only like to allow an admin (or superuser) to run something on a custom resource. However, I can live without this is there is a known issue.

Here is the section (that works) if I comment out the offending statements. The 'xxx' are entries I want to keep private:

profiles:
    launcher.kubernetes.profiles.conf:
      "*":
        default-cpus: 1
        default-mem-mb: 512
        container-images:
          - "xxx"
          - "xxx"
          - "xxx"
        default-container-image: "xxx"
        allow-unknown-images: 0 # Whether to allow users to bring their own image
        job-json-overrides:
          - target: "/spec/template/metadata/annotations/cluster-autoscaler.kubernetes.io~1safe-to-evict"
            json: "false"
            name: evict-annotation
        resource-profiles: A-ExtraSmall,B-Small,C-Medium
        # allow-custom-resources: 0
      "@superuser":
        resource-profiles: A-ExtraSmall,B-Small,C-Medium,D-Large,E-ExtraLarge
        # allow-custom-resources: 1
      "@admin":
        resource-profiles: A-ExtraSmall,B-Small,C-Medium,D-Large,E-ExtraLarge,F-Risk
        # allow-custom-resources: 1

Thanks!

[colearendt]

Howdy @murraj ! Thanks so much for reporting this. And thanks for your patience. This definitely feels like an issue / oversight in the product - I will pass this message along internally to see if we can reproduce it!