How to have $external_labels show up in discord?

[ki4rbc]

I am attempting to use an external label, locID, with Prometheus alerts. I have the external label being sent to alertmanager-discord, but how does one configure what's going to discord; other then modifying main.go?

Here is the print-pretty formatted input JSON:

{
	"receiver": "prometheus-msteams",
	"status": "firing",
	"alerts": [{
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-apiservers",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 kubernetes-apiservers jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:36:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-nodes",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 2 kubernetes-nodes jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-nodes-cadvisor",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 2 kubernetes-nodes-cadvisor jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "kubernetes-service-endpoints",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 kubernetes-service-endpoints jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:35:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}, {
		"status": "firing",
		"labels": {
			"alertname": "A test alert",
			"job": "prometheus",
			"locID": "00000000-0000-0000-0000-000000000000",
			"severity": "notice"
		},
		"annotations": {
			"description": "There are 1 prometheus jobs running",
			"summary": "This is only a test on locationID: "
		},
		"startsAt": "2020-08-31T18:36:25.881534544Z",
		"endsAt": "0001-01-01T00:00:00Z",
		"generatorURL": "http://prometheus-subscriber-server-6ccf748db-s4j9s:9090/graph?g0.expr=sum+by%28job%29+%28up%29+%3E+0\u0026g0.tab=1"
	}],
	"groupLabels": {
		"alertname": "A test alert"
	},
	"commonLabels": {
		"alertname": "A test alert",
		"locID": "00000000-0000-0000-0000-000000000000",
		"severity": "notice"
	},
	"commonAnnotations": {
		"summary": "This is only a test on locationID: "
	},
	"externalURL": "",
	"version": "4",
	"groupKey": "{}:{alertname=\"A test alert\"}"
}```

[benjojo]

Cheers for the input json context,

Right now there is not a way for you to do this, The way I do this in my infra is to inline the interesting data bits into my alerts description

What would you want to do with the locID? Just display it?

[ki4rbc]

Thanks for responding so quickly,
The locID is how I know where the alert is coming from it comes from an os env which is then used as a yaml anchor which is later used to create the external_label.

The problem I am running into is that external_labels are not available for processing within alert rules; unless I am missing something.

And yes. I would like locID to show up anywhere in the alert in discord.

[ki4rbc]

Hello again @benjojo,
So I got this mod of main.go to work but I'm not entirely happy and it's not something I think I would suggest a fork for.

--- original/alertmanager-discord/main.go       2020-08-31 18:35:49.698522600 -0400
+++ alertmanager-discord/main.go        2020-08-31 22:28:46.702549100 -0400
@@ -121,7 +121,7 @@
                                }

                                RichEmbed.Fields = append(RichEmbed.Fields, discordEmbedField{
-                                       Name:  fmt.Sprintf("[%s]: %s on %s", strings.ToUpper(status), alert.Labels["alertname"], realname),
+                                       Name:  fmt.Sprintf("[%s]: %s on %s - %s", strings.ToUpper(status), alert.Labels["alertname"], realname, alert.Labels["locID"]),
                                        Value: alert.Annotations.Description,
                                })
                        }

[benjojo]

Yeah I think for now a fork is the best, I kinda want to see if this is a one off use case or something other people have issues with too

…

On Tue, Sep 1, 2020 at 6:42 PM Mike McDonough ***@***.***> wrote: Hello again @benjojo <https://github.com/benjojo>, So I got this mod of main.go to work but I'm not entirely happy and it's not something I think I would suggest a fork for. --- original/alertmanager-discord/main.go 2020-08-31 18:35:49.698522600 -0400+++ alertmanager-discord/main.go 2020-08-31 22:28:46.702549100 -0400@@ -121,7 +121,7 @@ } RichEmbed.Fields = append(RichEmbed.Fields, discordEmbedField{- Name: fmt.Sprintf("[%s]: %s on %s", strings.ToUpper(status), alert.Labels["alertname"], realname),+ Name: fmt.Sprintf("[%s]: %s on %s - %s", strings.ToUpper(status), alert.Labels["alertname"], realname, alert.Labels["locID"]), Value: alert.Annotations.Description, }) } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALPK4XC5Q6NI7HOT6YMYDLSDUQATANCNFSM4QRAES7Q> .

[ki4rbc]

I found a work around to adding an alert origin identity, locationID in my case. By creating the alertmanager alerting rules is a config map template, I have access to environmental variables. It took a while to figure out I have to escape curly brackets. For example

          - alert: Deployments unavailable
            expr: 100 * (kube_deployment_status_replicas_available / kube_deployment_status_replicas) != 100
            for: 5m
            labels:
              severity: warn
            annotations:
              summary: 'Deployments unavailable at Location ID {{ .Values.locationID }}'
              description: '{{ "{{" }} $labels.deployment {{ "}}" }}.{{ "{{" }} $labels.namespace {{ "}}" }} at {{ "{{" }} $value | printf "%.2f" {{ "}}" }}% of intent'

I now can see where the alerts originate from.

I did see that on some alerts there is no label instance or exported_instance so we can still end up with alerts where realname is NULL.

I've created a fork and will work though a solution to propose.