How to remove node-taint via apiclient?

[makarov-roman]

Hi,

I have the following use case:

1. there is a taint by default on our nodes
2. bootstrap container fetches pullsecret
3. if it succeded remove the taint

the trouble is with the step 3, I can't find how to remove it from the config. Here are some examples of what I've tried:

apiclient set --json '{"kubernetes": {"node-taints": {"pullsecrets-not-ready": []}}}'
Failed to change settings: Failed PATCH request to '/settings?tx=apiclient-set-aFXHfCsm713hVmCu': Status 400 when PATCHing /settings?tx=apiclient-set-aFXHfCsm713hVmCu: Json deserialize error: empty taint value at line 1 column 57

apiclient set --json '{"kubernetes": {"node-taints": {}}}'
Failed to change settings: Failed POST request to '/tx/commit_and_apply?tx=apiclient-set-AB7MA1Cd92vVA8JA': Status 422 when POSTing /tx/commit_and_apply?tx=apiclient-set-AB7MA1Cd92vVA8JA: Tried to commit with no pending changes

Thanks!

[vigh-m]

Hi, Thanks reaching out,

Taints live on the Node object in the Kubernetes API server, so they can only be removed through the Kubernetes API itself.

Kubelet's --register-with-taints flag is only applied at initial node registration. On subsequent kubelet restarts, taints are not reconciled. Removing the taint from Bottlerocket's settings and restarting kubelet would have no effect on the actual node.

Alternatives

For your use case (bootstrap container fetches pull secret → remove taint on success), there are a few approaches you can try:

• DaemonSet with a toleration: Deploy a DaemonSet that tolerates pullsecrets-not-ready, checks that the pull secret is available, and then removes the taint from its own node via the Kubernetes API

• Host container: A Bottlerocket host container that waits for node registration, verifies the pull secret, and removes the taint via kubectl or the Kubernetes API directly. However, this would require the node to have wkubeconfig access and appropriate RBAC.

• Mutating admission webhook: Intercepts Node create/update events and can strip the taint at admission time based on your conditions.

• External automation: A controller, Lambda function, or CI/CD step that watches for node registration events and removes the taint via the kube API.

Hope that helps!

[makarov-roman]

thanks for the explanation and the suggestions.
I decided to use labels for now because it's the simplest solution. It's very easy to overengineer here.

• daemonSet can technically replace the bootstrap container at all, but I don't like the idea that we're always running a container which we need once a month (uses podsLimit)
• host container with k8s api might be the proper way, I'm considering it
• webhook is something I completely missed. e.g. it can react on label update and remove the taint. Sounds reasonable.