mimir.rules.kubernetes: namespace separator breaks with proxies that decode %2F

[TheoBrigitte]

Description

The mimir.rules.kubernetes component constructs Mimir namespace strings using / as a hardcoded separator in mimirNamespaceForRuleCRD():

return fmt.Sprintf("%s/%s/%s/%s", prefix, pr.Namespace, pr.Name, pr.UID)

This produces namespace values like:

operations/pagerduty-opsgenie-proxy/pagerduty-opsgenie-proxy/c835fbe6-6712-4321-a048-1a730a73c287

When Alloy syncs rules to the Mimir ruler API at /prometheus/config/v1/rules/{namespace}, the / characters in the namespace value get URL-encoded to %2F. Many reverse proxies decode %2F back to / before forwarding the request, which breaks path routing entirely.

Environment

• Alloy v1.12.0 (checked source up to v1.14.0 — separator is still hardcoded)
• Mimir behind Envoy Gateway v1.6.3
• Envoy's default escapedSlashesAction is UnescapeAndRedirect, which decodes %2F → /, resulting in a 307 redirect that resolves to a 404

This is not specific to Envoy. nginx (merge_slashes on / allow_special_chars) and Apache (AllowEncodedSlashes) have similar default behaviors.

Steps to reproduce

1. Deploy mimir.rules.kubernetes pointing at a Mimir instance behind Envoy Gateway (or any proxy with default encoded-slash handling)
2. Watch Alloy attempt to POST rule groups to /prometheus/config/v1/rules/operations%2Fmy-namespace%2Fmy-rulecrd%2F<uid>
3. Envoy decodes %2F and redirects — Alloy receives a 307 or 404

Expected behavior

Rules are synced successfully to Mimir regardless of whether a reverse proxy sits in front of it.

Actual behavior

Alloy fails to sync rules. Depending on proxy configuration, it receives a 307 redirect followed by a 404, or a direct 404.

Workaround

Configure the proxy to preserve encoded slashes. For Envoy Gateway, this means patching the ClientTrafficPolicy with:

spec:
  path:
    escapedSlashesAction: KeepUnchanged

This works but has two downsides:

• It requires control over the proxy configuration, which may not always be possible
• The policy applies to all routes on the gateway, not just the Mimir ruler endpoint

Proposal

Add a configurable mimir_namespace_separator argument to mimir.rules.kubernetes, defaulting to / for backward compatibility:

mimir.rules.kubernetes "default" {
  address                   = "http://mimir.example.com"
  mimir_namespace_separator = "_"
  ...
}

Using a separator that doesn't require URL encoding (e.g., _ or -) would sidestep the proxy issue entirely without requiring any infrastructure changes. The default of / keeps existing behavior unchanged for anyone not hitting this issue.

Related issues

• "Could not view rule" for alert rules provisioned via Grafana-Agent flow mode #260 (closed — the fix there was on the Grafana Cloud client side, not in Alloy itself)
• "Could not view rule" for alert rules provisioned via Grafana-Agent flow mode grafana#76628

[QuentinBisson]

@kalleep here is an attempt at fixing this #5961.

I deployed in in my cluster and it is running fine :)