Skip to content

fix: add OperatorConfig enableMatchOneOf flag for stuck match #1810

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bwplotka wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: add OperatorConfig enableMatchOneOf flag for stuck match #1810

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 34 additions & 3 deletions charts/operator/crds/monitoring.googleapis.com_operatorconfigs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,12 +90,43 @@ spec:
description: Filter limits which metric data is sent to Cloud Monitoring
(it doesn't apply to additional exports).
properties:
enableMatchOneOf:
description: |-
EnableMatchOneOf allows additional control over MatchOneOf filtering.

Available settings:
* not set: The MatchOneOf settings are ignored, default is used.
* false: MatchOneOf feature is explicitly disabled; export is forced to match all series.
* true: The MatchOneOf settings are used, overwriting a default.

See MatchOneOf IMPORTANT section to learn about the MatchOneOf default.
type: boolean
matchOneOf:
description: |-
A list of Prometheus time series matchers. Every time series must match at least one
of the matchers to be exported. This field can be used equivalently to the match[]
parameter of the Prometheus federation endpoint to selectively export data.
MatchOneOf, if EnableMatchOneOf is true, controls the export filtering setting.

MatchOneOf expects a list of Prometheus time series matchers. Every time series
must match at least one of the matchers to be exported. This field can be used
equivalently to the match[] parameter of the Prometheus federation endpoint to
selectively export data.

Example: `["{job!='foobar'}", "{__name__!~'container_foo.*|container_bar.*'}"]`

IMPORTANT: MatchOneOf is guarded by the additional flag (EnableMatchOneOf)
and removed from the public docs. Replacements: https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#filter-metrics.

Rationales:
* This option is prone to misconfiguration, e.g. you need to manually
match important built-in metrics (up, scrape_samples_added, etc.).
* Unmatched metrics are still in the local collector memory
(this filters on export only).
* Typically, the default (empty array), means match all metrics. However,
for the clusters that were using MatchOneOf filtering before the GMP 0.14.x,
version, the default is what has been configured back then in the orphaned
collector DaemonSet EXTRA_ARGS environment variable. If you are impacted:
* use EnableMatchOneOf = false to reset the default
* use EnableMatchOneOf = true to apply this configuration's MatchOneOf
* Remove EXTRA_ARGS environment variable from the collector DaemonSet (it's safe to do so).
items:
type: string
type: array
Expand Down
18 changes: 9 additions & 9 deletions charts/values.global.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,30 +16,30 @@ commonLabels: false
namespace:
public: gmp-public
system: gmp-system
version: 0.15.4
version: 0.17.2
images:
# NOTE: All tags have to be quoted otherwise they might be treated as a number.
bash:
image: gke.gcr.io/gke-distroless/bash
tag: "gke_distroless_20250407.00_p0"
tag: gke_distroless_20251107.00_p0
alertmanager:
image: gke.gcr.io/prometheus-engine/alertmanager
tag: "v0.27.0-gmp.3-gke.0"
tag: "v0.27.0-gmp.4-gke.4"
prometheus:
image: gke.gcr.io/prometheus-engine/prometheus
tag: "v2.53.4-gmp.0-gke.1"
tag: "v2.53.5-gmp.1-gke.2"
configReloader:
image: gke.gcr.io/prometheus-engine/config-reloader
tag: "v0.15.4-gke.0"
tag: "v0.17.2-gke.2"
operator:
image: gke.gcr.io/prometheus-engine/operator
tag: "v0.15.4-gke.0"
tag: "v0.17.2-gke.2"
ruleEvaluator:
image: gke.gcr.io/prometheus-engine/rule-evaluator
tag: "v0.15.4-gke.0"
tag: "v0.17.2-gke.2"
datasourceSyncer:
image: gke.gcr.io/prometheus-engine/datasource-syncer
tag: "v0.15.4-gke.0"
tag: "v0.17.2-gke.2"
resources:
alertManager:
limits:
Expand Down Expand Up @@ -98,4 +98,4 @@ ruleEvaluator:
create: true
serviceAccount:
create: true
name: rule-evaluator
name: rule-evaluator
4 changes: 2 additions & 2 deletions cmd/datasource-syncer/datasource-syncer.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ spec:
- linux
containers:
- name: datasource-syncer-init
image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.15.4-gke.0
image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.17.2-gke.2
args:
- "--datasource-uids=$DATASOURCE_UIDS"
- "--grafana-api-endpoint=$GRAFANA_API_ENDPOINT"
Expand Down Expand Up @@ -79,7 +79,7 @@ spec:
- linux
containers:
- name: datasource-syncer
image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.15.4-gke.0
image: gke.gcr.io/prometheus-engine/datasource-syncer:v0.17.2-gke.2
args:
- "--datasource-uids=$DATASOURCE_UIDS"
- "--grafana-api-endpoint=$GRAFANA_API_ENDPOINT"
Expand Down
8 changes: 0 additions & 8 deletions cmd/rule-evaluator/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,14 +80,6 @@ Flags:
targets. Prefer setting the external label
"cluster" in the Prometheus configuration if
not using the auto-discovered default.
--export.match= ... A Prometheus time series matcher. Can be
repeated. Every time series must match at
least one of the matchers to be exported.
This flag can be used equivalently to the
match[] parameter of the Prometheus federation
endpoint to selectively export data.
(Example: --export.match='{job="prometheus"}'
--export.match='{__name__=~"job:.*"})
--export.debug.metric-prefix="prometheus.googleapis.com"
Google Cloud Monitoring metric prefix to use.
--[no-]export.debug.disable-auth
Expand Down
40 changes: 36 additions & 4 deletions doc/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1009,16 +1009,48 @@ ClusterPodMonitoring, PodMonitoring, GlobalRules, ClusterRules, and/or Rules.</p
<tbody>
<tr>
<td>
<code>enableMatchOneOf</code><br/>
<em>
bool
</em>
</td>
<td>
<p>EnableMatchOneOf allows additional control over MatchOneOf filtering.</p>
<p>Available settings:
* not set: The MatchOneOf settings are ignored, default is used.
* false: MatchOneOf feature is explicitly disabled; export is forced to match all series.
* true: The MatchOneOf settings are used, overwriting a default.</p>
<p>See MatchOneOf IMPORTANT section to learn about the MatchOneOf default.</p>
</td>
</tr>
<tr>
<td>
<code>matchOneOf</code><br/>
<em>
[]string
</em>
</td>
<td>
<p>A list of Prometheus time series matchers. Every time series must match at least one
of the matchers to be exported. This field can be used equivalently to the match[]
parameter of the Prometheus federation endpoint to selectively export data.
Example: <code>[&quot;{job!='foobar'}&quot;, &quot;{__name__!~'container_foo.*|container_bar.*'}&quot;]</code></p>
<p>MatchOneOf, if EnableMatchOneOf is true, controls the export filtering setting.</p>
<p>MatchOneOf expects a list of Prometheus time series matchers. Every time series
must match at least one of the matchers to be exported. This field can be used
equivalently to the match[] parameter of the Prometheus federation endpoint to
selectively export data.</p>
<p>Example: <code>[&quot;{job!='foobar'}&quot;, &quot;{__name__!~'container_foo.*|container_bar.*'}&quot;]</code></p>
<p>IMPORTANT: MatchOneOf is guarded by the additional flag (EnableMatchOneOf)
and removed from the public docs. Replacements: <a href="https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#filter-metrics">https://cloud.google.com/stackdriver/docs/managed-prometheus/setup-managed#filter-metrics</a>.</p>
<p>Rationales:
* This option is prone to misconfiguration, e.g. you need to manually
match important built-in metrics (up, scrape_samples_added, etc.).
* Unmatched metrics are still in the local collector memory
(this filters on export only).
* Typically, the default (empty array), means match all metrics. However,
for the clusters that were using MatchOneOf filtering before the GMP 0.14.x,
version, the default is what has been configured back then in the orphaned
collector DaemonSet EXTRA_ARGS environment variable. If you are impacted:
* use EnableMatchOneOf = false to reset the default
* use EnableMatchOneOf = true to apply this configuration&rsquo;s MatchOneOf
* Remove EXTRA_ARGS environment variable from the collector DaemonSet (it&rsquo;s safe to do so).</p>
</td>
</tr>
</tbody>
Expand Down
Loading
Loading