ClientTrafficPolicy: caCertificateRefs foreign namespaces not supported

[BernhardGruen]

Description
I tried to configure an mTLS secret for client connections. I wanted to use a secret from another namespace. Therefore I created a ReferenceGrant for it.
But unfortunately this did not work even though it is documented as possible: https://gateway.envoyproxy.io/docs/api/extension_types/#clientvalidationcontext

References to a resource in different namespace are invalid UNLESS there is a ReferenceGrant in the target namespace that allows the certificate to be attached.

It seems the code that handles the ClientTrafficPolicy does not allow foreign namespaces as the following code shows:

gateway/internal/gatewayapi/clienttrafficpolicy.go

Line 901 in 344a9d8

secret, err := t.validateSecretRef(false, from, ref, resources)

The first parameter would have to be true in this case to allow foreign namespaces. I don't know the software therefore I am not sure if this is the only place.

It would be great if a future version would support this case.

Repro steps
Create the following objects:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: gw-eg
  namespace: test-01
spec:
  gatewayClassName: eg
  listeners:
    - allowedRoutes:
        namespaces:
          from: Same
      hostname: "test-01.example.eu"
      name: http
      protocol: HTTP
      port: 80
    - allowedRoutes:
        namespaces:
          from: Same
      hostname: "test-01.example.eu"
      name: https
      port: 443
      protocol: HTTPS
      tls:
        certificateRefs:
        - group: ""
          kind: Secret
          name: tls-sec
        mode: Terminate
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-httproute-eg
  namespace: test-01
spec:
  hostnames:
  - "test-01.example.eu"
  parentRefs:
    - name: gw-eg
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: my-service
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: ClientTrafficPolicy
metadata:
  name: enable-mtls
  namespace: test-01
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: gw-eg
  tls:
    clientValidation:
      optional: false

      caCertificateRefs:
      - kind: "Secret"
        group: ""
        namespace: pki
        name: "ca-client-auth-secret"
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: client-auth-secret
  namespace: pki
spec:
  from:
  - group: gateway.envoyproxy.io
    kind: ClientTrafficPolicy
    namespace: test-01
  to:
  - group: ""
    kind: Secret
    name: "ca-client-auth-secret"

As you can see the ReferenceGrant was generated in the pki namespace as it is written in the documentation.

Environment
Kubernetes 1.33.4, Envoy Gateway 1.6.1, Gateway-API 1.4.1 (standard)

Logs
This error message is shown in the logs and also in the status of ClientTrafficPolicy:

TLS: failed to get certificate from ref: secret ref namespace must  be unspecified/empty or test-01

[arkodg]

cc @rudrakhp @zhaohuabing

[filleokus]

Just want to add I got the same error with a SecurityPolicy using OIDC with a client secret from another namespace.

"OIDC: secret ref namespace must be unspecified/empty or target-namespace"

It is not as explicitly documented to be supported as mTLS however, but would be nice if it was supported :)

Relevant manifests:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  annotations:
  generation: 1
  name: example-policy
  namespace: target-namespace
spec:
  authorization:
    defaultAction: Deny
    rules:
    - action: Allow
      name: allow-example-admins
      principal:
        jwt:
          claims:
          - name: groups
            valueType: StringArray
            values:
            - c76b543f-193a-4989-965d-24ad7eba4205
          provider: entra-id
  jwt:
    providers:
    - extractFrom:
        cookies:
        - IdToken
        headers:
        - name: Authorization
          valuePrefix: 'Bearer '
      issuer: https://auth.example.com
      name: entra-id
      remoteJWKS:
        cacheDuration: 300s
        uri: https://auth.example.com/discovery/v2.0/keys
  oidc:
    clientSecret:
      group: ""
      kind: Secret
      name: oidc-secret
      namespace: source-namespace
    clientID: example-client-id
    cookieNames:
      accessToken: AccessToken
      idToken: IdToken
    disableTokenEncryption: true
    passThroughAuthHeader: true
    provider:
      issuer: https://auth.example.com
    refreshToken: true
    scopes:
    - openid
    - email
    - profile
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: example-route
---
kind: ReferenceGrant
metadata:
  name: allow-oidc-secret-target-namespace
  namespace: source-namespace
spec:
  from:
  - group: gateway.envoyproxy.io
    kind: SecurityPolicy
    namespace: target-namespace
  to:
  - group: ""
    kind: Secret
    name: oidc-secret

[zhaohuabing]

Just want to add I got the same error with a SecurityPolicy using OIDC with a client secret from another namespace.

"OIDC: secret ref namespace must be unspecified/empty or target-namespace"

It is not as explicitly documented to be supported as mTLS however, but would be nice if it was supported :)

Hi @filleokus could you please explain your use case? Why do you need to put the client secret in a different namespace?

[filleokus]

For some context: We host both public facing and internal applications in our cluster. We have two gateways, one for public facing and one for internal applications. On the internal gateway we currently have a SecurityPolicy that just requires the user to be authenticated with our internal IdP.

For certain internal apps, we now want to have SecurityPolicies that overrides the base SecurityPolicy on that specific HTTPRoute, specifying additional requirements on jwt.claims (like that only the admin group is authorized to use it). Ideally we would like to "delegate" the management of the jwt.claims field to the application namespaces and avoid copy-pasting the OIDC section. This since the application namespaces are managed by other teams than the central Envoy config.

Our idea to solve this was to have a kustomize component with the base security policy and then render it out in the envoy-gateway-systems namespace, overriding name/ns. In the app namespaces we would do the same but patching in the spec.authorization.rules section with additional requirements.

If we could use reference grants for the client secret this would work. If not, we would need to have some mechanism that deploys the client secret in to the app namespaces, or give up on the "delegation" part and just control it centrally.

It's not a huge issue, but seems like not just us could have a use case for something like this 😄

[tszyszko]

Similar issue and use case to @filleokus . Considering using secret reflector to copy the client secret into each app namespace, but reference grant is a much neater solution if possible

[zhaohuabing]

hi @ filleokus it sounds like the new mergeType feature would help your use case - the app team can define a Route level authorization rule and merge it with a Gateway/Listener level authentication rule.

[filleokus]

Ah! Wonderful news, for our use-case this is seems even better, we can get rid of the kustomize machinery to do the merge ❤️

[zhaohuabing]

Discussed in today’s community meeting: the decision is to support the cross-namespace for SecretObjectReferences in the existing API.