ResponseOverride ConfigMap references break with merged BackendTrafficPolicies in multiple namespaces

[bitlkoskela]

Description:

With merged BackendTrafficPolicies, where the policies exist in different namespaces, responseOverride with a ConfigMap reference may break after merging. This causes routes to be set to return 500 and this is seen in logs.

gatewayapi/backendtrafficpolicy.go:837  setting 500 direct response in routes due to errors in BackendTrafficPolicy {"runner": "gateway-api", "trace_id": "d4007bcdcfa56c7239bd4888522156ad", "span_id": "050d3338a03a1d10", "policy": {"name":"my-app","namespace":"my-app"}, "routes": ["grpcroute/my-app/my-app/rule/0/match/-1/myapp_example_com"], "error": "ResponseOverride: can't find the referenced configmap error-page"}

I assume that after merging BackendTrafficPolicies, the ConfigMap reference is handled in the wrong namespace. There's no way to specify the namespace in the responseOverride ValueRef.

Repro steps:
Create these BackendTrafficPolicies, ConfigMap and HTTPRoute. Note the different namespaces. Observe 500 responses from Envoy, with "can't find the referenced configmap" in the logs.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: my-gateway-error-response
  namespace: envoy-gateway-system
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway
  responseOverride:
  - match:
      statusCodes:
      - type: Range
        range:
          start: 502
          end: 504
      response:
        contentType: "text/html"
        body:
          type: "ValueRef"
          valueRef:
            group: ""
            kind: ConfigMap
            name: error-page

apiVersion: v1
kind: ConfigMap
metadata:
  name: error-page
  namespace: envoy-gateway-system
data:
  response.body: |
    error page contents here

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: my-app-rate-limit
  namespace: my-app
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: my-app
  mergeType: StrategicMerge
  rateLimit:
    local:
      rules:
      - clientSelectors:
        - sourceCIDR:
            type: Distinct
            value: 0.0.0.0/0
        limit:
          requests: 200
          unit: Minute

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: my-app
  namespace: my-app
spec:
  hostnames:
  - myapp.example.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: my-gateway
    namespace: envoy-gateway-system
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: my-app
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /

Environment:
Envoy gateway 1.7.0

Logs:

gatewayapi/backendtrafficpolicy.go:837  setting 500 direct response in routes due to errors in BackendTrafficPolicy {"runner": "gateway-api", "trace_id": "d4007bcdcfa56c7239bd4888522156ad", "span_id": "050d3338a03a1d10", "policy": {"name":"my-app","namespace":"my-app"}, "routes": ["grpcroute/my-app/my-app/rule/0/match/-1/myapp_example_com"], "error": "ResponseOverride: can't find the referenced configmap error-page"}

[bitlkoskela]

Another detail I noticed: this error state seems to not be shown in the object status fields. All BackendTrafficPolicies are shown as Merged and Accepted, and HTTPRoute as Accepted. Still EG detects the error and sets forces the routes to return 500? Should this be visible in BTP status? Where could I detect this situation except the logs?

[arkodg]

cc @rudrakhp

[rudrakhp]

/assign