Prometheus

Query for in-use deprecated APIs

If prometheus scrapes the k8s metrics (https://kubernetes.io/docs/reference/instrumentation/metrics/):

apiserver_requested_deprecated_apis

Slow rule evaluations

When PrometheusMissingRuleEvaluations alert hits, some group took more than the evaluation_interval.

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/#failed-rule-evaluations-due-to-slow-evaluation

Find slow groups:

prometheus_rule_group_last_duration_seconds > 1

Rule health:

In the Prometheus UI under "Rule Health" you'll find the duration and health of each rule in each rule group. This information is not exposed as a metric.

Number of samples scraped

rule evaluation duration can be affected by the number of samples to go through

scrape_samples_scraped and scrape_samples_post_metric_relabeling can help here.

https://prometheus.io/docs/concepts/jobs_instances/