Skip to content

Commit af8f5fe

Browse files
schahalclaudediranged
authored
fix: replace deprecated kube-rbac-proxy with controller-runtime authn/authz (#443)
## Background The `gcr.io/kubebuilder/kube-rbac-proxy` image is deprecated and Google Container Registry (GCR) is being sunset, making the image unavailable. This project used kube-rbac-proxy as a sidecar container to secure the `/metrics` endpoint via Kubernetes TokenReview and SubjectAccessReview. See [kubernetes-sigs/cluster-api-addon-provider-helm#318](kubernetes-sigs/cluster-api-addon-provider-helm#318) for upstream context. ## Changes Instead of finding an alternative image, this replaces the sidecar pattern entirely with controller-runtime's built-in `SecureServing` and `WithAuthenticationAndAuthorization` filter. The manager now serves metrics securely over HTTPS on port 8443 with native authn/authz — no sidecar needed. The kube-rbac-proxy container, its image configuration, and related Helm values have been removed from both the kustomize config and the Helm chart. The existing RBAC resources (TokenReview/SubjectAccessReview permissions and metrics-reader ClusterRole) are retained since the manager now performs these checks itself. ## Testing Deployed to a local KIND cluster and verified the pod runs with a single container (no sidecar). Confirmed that unauthenticated requests to the metrics endpoint return `Unauthorized`. Verified that authenticated requests without the `metrics-reader` ClusterRole return `Authorization denied`. Confirmed that authenticated requests with the `metrics-reader` role bound successfully return Prometheus metrics. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Matt Wise <matt@nextdoor.com>
1 parent 673f231 commit af8f5fe

16 files changed

Lines changed: 127 additions & 93 deletions

‎charts/oz/README.md‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -32,12 +32,6 @@ Kubernetes: `>=1.26.0-0`
3232

3333
| Key | Type | Default | Description |
3434
|-----|------|---------|-------------|
35-
| controllerManager.kubeRbacProxy.image.repository | string | `"gcr.io/kubebuilder/kube-rbac-proxy"` | |
36-
| controllerManager.kubeRbacProxy.image.tag | string | `"v0.16.0"` | |
37-
| controllerManager.kubeRbacProxy.resources.limits.cpu | string | `"500m"` | |
38-
| controllerManager.kubeRbacProxy.resources.limits.memory | string | `"128Mi"` | |
39-
| controllerManager.kubeRbacProxy.resources.requests.cpu | string | `"5m"` | |
40-
| controllerManager.kubeRbacProxy.resources.requests.memory | string | `"64Mi"` | |
4135
| controllerManager.manager.image.repository | `string` | `"ghcr.io/diranged/oz"` | Docker Image repository and name to use for the controller. |
4236
| controllerManager.manager.image.tag | `string` | `nil` | If set, overrides the .Chart.AppVersion field to set the target image version for the Oz controller. |
4337
| controllerManager.manager.resources.limits.cpu | string | `"500m"` | |

‎charts/oz/templates/deployment.yaml‎

Lines changed: 14 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -54,40 +54,13 @@ spec:
5454
secretName: {{ . }}
5555
{{- end }}
5656
containers:
57-
- name: kube-rbac-proxy
58-
image: {{ .Values.controllerManager.kubeRbacProxy.image.repository }}:{{ .Values.controllerManager.kubeRbacProxy.image.tag }}
59-
securityContext:
60-
allowPrivilegeEscalation: false
61-
capabilities:
62-
drop: [ALL]
63-
args:
64-
- --secure-listen-address=0.0.0.0:8443
65-
- --upstream=http://127.0.0.1:8080/
66-
- --logtostderr=true
67-
- --v=0
68-
env:
69-
- name: KUBERNETES_CLUSTER_DOMAIN
70-
value: {{ .Values.kubernetesClusterDomain }}
71-
ports:
72-
{{- /* Default Metrics Endpoint */}}
73-
{{- with (index .Values.metricsService.ports 0) }}
74-
- containerPort: 8443
75-
name: {{ .targetPort }}
76-
protocol: {{ .protocol }}
77-
{{- end }}
78-
79-
{{- with .Values.controllerManager.kubeRbacProxy.resources }}
80-
resources:
81-
{{- toYaml . | nindent 10 }}
82-
{{- end }}
83-
8457
- name: manager
8558
image: {{ .Values.controllerManager.manager.image.repository }}:{{ .Values.controllerManager.manager.image.tag | default .Chart.Version .Chart.AppVersion }}
8659
command:
8760
- /manager
8861
args:
8962
- --health-probe-bind-address=:8081
90-
- --metrics-bind-address=127.0.0.1:8080
63+
- --metrics-bind-address=:8443
9164
- --leader-elect
9265
securityContext:
9366
allowPrivilegeEscalation: false
@@ -113,18 +86,23 @@ spec:
11386
{{- toYaml . | nindent 10 }}
11487
{{- end }}
11588

89+
ports:
90+
{{- /* Metrics Endpoint */}}
91+
- containerPort: 8443
92+
name: https
93+
protocol: TCP
94+
95+
{{- /* Optional Webhook Endpoint */}}
96+
{{- with (index .Values.webhookService.ports 0) }}
97+
- containerPort: 9443
98+
name: {{ .targetPort }}
99+
protocol: {{ .protocol }}
100+
{{- end }}
101+
116102
{{- /* Optional Webhook Endpoint */}}
117103
{{- with .Values.webhook.secret.name }}
118104
volumeMounts:
119105
- mountPath: /tmp/k8s-webhook-server/serving-certs
120106
name: cert
121107
readOnly: true
122108
{{- end }}
123-
124-
{{- /* Optional Webhook Endpoint */}}
125-
{{- with (index .Values.webhookService.ports 0) }}
126-
ports:
127-
- containerPort: 9443
128-
name: {{ .targetPort }}
129-
protocol: {{ .protocol }}
130-
{{- end }}

‎charts/oz/templates/metrics-reader-rbac.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: ClusterRole
33
metadata:
44
name: {{ include "oz.fullname" . }}-metrics-reader
55
labels:
6-
app.kubernetes.io/component: kube-rbac-proxy
6+
app.kubernetes.io/component: metrics
77
app.kubernetes.io/created-by: oz
88
app.kubernetes.io/instance: metrics-reader
99
app.kubernetes.io/name: clusterrole

‎charts/oz/templates/metrics-service.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: Service
33
metadata:
44
name: {{ include "oz.fullname" . }}-controller-manager-metrics-service
55
labels:
6-
app.kubernetes.io/component: kube-rbac-proxy
6+
app.kubernetes.io/component: metrics
77
app.kubernetes.io/created-by: oz
88
app.kubernetes.io/instance: controller-manager-metrics-service
99
app.kubernetes.io/name: service

‎charts/oz/templates/proxy-rbac.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: ClusterRole
33
metadata:
44
name: {{ include "oz.fullname" . }}-proxy-role
55
labels:
6-
app.kubernetes.io/component: kube-rbac-proxy
6+
app.kubernetes.io/component: metrics
77
app.kubernetes.io/created-by: oz
88
app.kubernetes.io/instance: proxy-role
99
app.kubernetes.io/name: clusterrole
@@ -28,7 +28,7 @@ kind: ClusterRoleBinding
2828
metadata:
2929
name: {{ include "oz.fullname" . }}-proxy-rolebinding
3030
labels:
31-
app.kubernetes.io/component: kube-rbac-proxy
31+
app.kubernetes.io/component: metrics
3232
app.kubernetes.io/created-by: oz
3333
app.kubernetes.io/instance: proxy-rolebinding
3434
app.kubernetes.io/name: clusterrolebinding

‎charts/oz/templates/webhook-service.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ kind: Service
33
metadata:
44
name: {{ include "oz.fullname" . }}-controller-manager-webhook-service
55
labels:
6-
app.kubernetes.io/component: kube-rbac-proxy
6+
app.kubernetes.io/component: webhook
77
{{- include "oz.labels" . | nindent 4 }}
88
spec:
99
type: {{ .Values.webhookService.type }}

‎charts/oz/values.yaml‎

Lines changed: 0 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -29,17 +29,6 @@ controllerManager:
2929
cpu: 10m
3030
memory: 64Mi
3131

32-
kubeRbacProxy:
33-
image:
34-
repository: gcr.io/kubebuilder/kube-rbac-proxy
35-
tag: v0.16.0
36-
resources:
37-
limits:
38-
cpu: 500m
39-
memory: 128Mi
40-
requests:
41-
cpu: 5m
42-
memory: 64Mi
4332

4433
# Configuration for the oz-controller-manager-metrics-service, used for
4534
# collecting metrics from the controller.

‎config/default/manager_auth_proxy_patch.yaml‎

Lines changed: 9 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
1-
# This patch inject a sidecar container which is a HTTP proxy for the
2-
# controller manager, it performs RBAC authorization against the Kubernetes API using SubjectAccessReviews.
1+
# This patch configures the manager to serve metrics securely using
2+
# controller-runtime's built-in authn/authz (SecureServing +
3+
# WithAuthenticationAndAuthorization), replacing the deprecated
4+
# kube-rbac-proxy sidecar.
35
apiVersion: apps/v1
46
kind: Deployment
57
metadata:
@@ -25,32 +27,13 @@ spec:
2527
values:
2628
- linux
2729
containers:
28-
- name: kube-rbac-proxy
29-
securityContext:
30-
allowPrivilegeEscalation: false
31-
capabilities:
32-
drop:
33-
- "ALL"
34-
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0
35-
args:
36-
- "--secure-listen-address=0.0.0.0:8443"
37-
- "--upstream=http://127.0.0.1:8080/"
38-
- "--logtostderr=true"
39-
- "--v=0"
40-
ports:
41-
- containerPort: 8443
42-
protocol: TCP
43-
name: https
44-
resources:
45-
limits:
46-
cpu: 500m
47-
memory: 128Mi
48-
requests:
49-
cpu: 5m
50-
memory: 64Mi
5130
- name: manager
5231
args:
5332
- "--health-probe-bind-address=:8081"
54-
- "--metrics-bind-address=127.0.0.1:8080"
33+
- "--metrics-bind-address=:8443"
5534
- "--leader-elect"
5635
- "--zap-log-level=5"
36+
ports:
37+
- containerPort: 8443
38+
name: https
39+
protocol: TCP

‎config/rbac/auth_proxy_client_clusterrole.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
labels:
55
app.kubernetes.io/name: clusterrole
66
app.kubernetes.io/instance: metrics-reader
7-
app.kubernetes.io/component: kube-rbac-proxy
7+
app.kubernetes.io/component: metrics
88
app.kubernetes.io/created-by: oz
99
app.kubernetes.io/part-of: oz
1010
app.kubernetes.io/managed-by: kustomize

‎config/rbac/auth_proxy_role.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
labels:
55
app.kubernetes.io/name: clusterrole
66
app.kubernetes.io/instance: proxy-role
7-
app.kubernetes.io/component: kube-rbac-proxy
7+
app.kubernetes.io/component: metrics
88
app.kubernetes.io/created-by: oz
99
app.kubernetes.io/part-of: oz
1010
app.kubernetes.io/managed-by: kustomize

0 commit comments

Comments
 (0)