Skip to content

Add toleration for prepull daemonset #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
amdove wants to merge 3 commits into from
Closed

Add toleration for prepull daemonset #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
147cd84
Add toleration for prepull daemonset
amdove Jan 29, 2026
e60bc46
Improved scheduling of prepull daemonset
amdove Jan 29, 2026
6de896e
Revert prepullnode plan, implement anti-affinity for system nodes
amdove Jan 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 50 additions & 23 deletions dist/chart/templates/crd/core.posit.team_sites.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,32 +1,11 @@
{{- if .Values.crd.enable }}
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
labels:
{{- include "chart.labels" . | nindent 4 }}
annotations:
{{- if .Values.certmanager.enable }}
cert-manager.io/inject-ca-from: "{{ .Release.Namespace }}/serving-cert"
{{- end }}
{{- if .Values.crd.keep }}
"helm.sh/resource-policy": keep
{{- end }}
controller-gen.kubebuilder.io/version: v0.17.0
name: sites.core.posit.team
spec:
{{- if .Values.webhook.enable }}
conversion:
strategy: Webhook
webhook:
clientConfig:
service:
namespace: {{ .Release.Namespace }}
name: webhook-service
path: /convert
conversionReviewVersions:
- v1
{{- end }}
group: core.posit.team
names:
kind: Site
Expand Down Expand Up @@ -486,6 +465,11 @@ spec:
flightdeck:
description: Flightdeck contains Flightdeck configuration
properties:
enabled:
description: |-
Enabled controls whether Flightdeck is deployed. Defaults to true if not specified.
Set to false to explicitly disable Flightdeck deployment.
type: boolean
featureEnabler:
description: FeatureEnabler controls which features are enabled
in Flightdeck
Expand All @@ -500,7 +484,11 @@ spec:
type: boolean
type: object
image:
description: Image is the container image for Flightdeck
description: |-
Image is the container image for Flightdeck.
Can be a tag (e.g., "v1.2.3") which will be combined with the default registry,
or a full image path (e.g., "my-registry.io/flightdeck:v1.0.0").
Defaults to "docker.io/posit/ptd-flightdeck:latest" if not specified.
type: string
imagePullPolicy:
description: ImagePullPolicy controls when the kubelet pulls the
Expand Down Expand Up @@ -1328,6 +1316,46 @@ spec:
description: SessionInitContainerImageTag specifies the init container
image tag for Workbench sessions
type: string
sessionTolerations:
description: SessionTolerations are tolerations applied only to
session pods (not the main workbench server)
items:
description: |-
The pod this Toleration is attached to tolerates any taint that matches
the triple <key,value,effect> using the matching operator <operator>.
properties:
effect:
description: |-
Effect indicates the taint effect to match. Empty means match all taint effects.
When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
type: string
key:
description: |-
Key is the taint key that the toleration applies to. Empty means match all taint keys.
If the key is empty, operator must be Exists; this combination means to match all values and all keys.
type: string
operator:
description: |-
Operator represents a key's relationship to the value.
Valid operators are Exists and Equal. Defaults to Equal.
Exists is equivalent to wildcard for value, so that a pod can
tolerate all taints of a particular category.
type: string
tolerationSeconds:
description: |-
TolerationSeconds represents the period of time the toleration (which must be
of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default,
it is not set, which means tolerate the taint forever (do not evict). Zero and
negative values will be treated as 0 (evict immediately) by the system.
format: int64
type: integer
value:
description: |-
Value is the taint value the toleration matches to.
If the operator is Exists, the value should be empty, otherwise just a regular string.
type: string
type: object
type: array
snowflake:
properties:
accountId:
Expand Down Expand Up @@ -1459,4 +1487,3 @@ spec:
storage: true
subresources:
status: {}
{{- end -}}
29 changes: 23 additions & 6 deletions internal/controller/core/site_controller_pre_pull.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,13 +104,30 @@ func deployPrePullDaemonset(ctx context.Context, r *SiteReconciler, req controll
},
}

if len(site.Spec.Workbench.Tolerations) > 0 {
// add the tolerations to the daemonset
for _, t := range site.Spec.Workbench.Tolerations {
prePullDaemonset.Spec.Template.Spec.Tolerations = append(prePullDaemonset.Spec.Template.Spec.Tolerations, *t.DeepCopy())
}
// Add universal toleration to run on all nodes regardless of taints
prePullDaemonset.Spec.Template.Spec.Tolerations = []v1.Toleration{
{
Operator: v1.TolerationOpExists,
},
}

// TODO: should also use the workbench node selectors...? But could differ from Connect...
// Add anti-affinity to avoid scheduling on system nodes (nodes labeled with posit.team/node-role: system)
prePullDaemonset.Spec.Template.Spec.Affinity = &v1.Affinity{
NodeAffinity: &v1.NodeAffinity{
RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{
NodeSelectorTerms: []v1.NodeSelectorTerm{
{
MatchExpressions: []v1.NodeSelectorRequirement{
{
Key: "posit.team/node-role",
Operator: v1.NodeSelectorOpNotIn,
Values: []string{"system"},
},
},
},
},
},
},
}
return nil
}); err != nil {
Expand Down