Add support for dogstatsd events

[russorat]

Feature Request

Proposal:

the current statsd input plugin has the ability to parse data dog tags. dogstatsd clients also have the ability to send events in a specific datadog format. https://docs.datadoghq.com/developers/dogstatsd/data_types/#events

This would either add the ability to parse those specific datadog events in the current statsd plugin, or move everything into a specific dogstatsd input plugin.

Current behavior:

datadog events are not parsed correctly from the statsd input plugin

Desired behavior:

either the current statsd plugin accepts and parses those events, or a new plugin is created specifically for dogstatsd

Use case: [Why is this important (helps with prioritizing requests)]

this is helpful for customers that want to send data to multiple monitoring solutions without having to change their existing application code using dogstatsd client code.

[danielnelson]

From on the Events API documentation, the example event is:

{
      "title": "Did you hear the news today?",
      "text": "Oh boy!",
      "priority": "normal",
      "tags": ["environment:test"],
      "alert_type": "info"
}

This could be received using the http_listener_v2 input configured with the json parser and to collect all the string fields. This could then be sent to InfluxDB or other outputs that support a string type, so maybe we just need to document a configuration example.

If we wanted to support send this type of metric to Datadog we would need a Datadog Events output.

[russorat]

@danielnelson i think the format for an event sent by dogstatsd client is slightly different. See the java client code here: https://github.com/DataDog/java-dogstatsd-client/blob/master/src/main/java/com/timgroup/statsd/NonBlockingStatsDClient.java#L921

[danielnelson]

Got it. I couldn't find any specification for this format but this appears to be the parser for Datadog events in statsd extension: https://github.com/DataDog/datadog-agent/blob/master/pkg/dogstatsd/parser.go#L152

[schahal]

@danielnelson, exactly! that's the parser of the Events datagram :) I submitted this PR so that those functions become exportable: DataDog/datadog-agent#2839

In the meantime, I added this PR on my own fork (I don't want to PR that branch to influxdata/telegraf:master until the exportable functions are merged by DD above and dependencies can be resolved).

That is, in final commit, we can change:

import (
    ...
    "github.com/schahal/datadog-agent/pkg/dogstatsd/parser"
    ...

to:

import (
    ...
    "github.com/DataDog/datadog-agent/pkg/dogstatsd/parser"
    ...

See schahal#1 for proposed implementation

Implementation Notes

We add support for the following 3 configuration keys in [[inputs.statsd]]:

  ## Parses events in the datadog statsd format to statsd protocol,
  ## by making 'title' and 'text' fields and the rest as tags.
  ## parse_data_dog_tags must also be set to true for this.
  ## https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/#events
  parse_data_dog_events = false

  ## If set, value of this event tag will be used as measurement name
  event_measurement_name_tag = "measurement_name"

  ## Datadog events will fallback/default to this measurement name
  ##   if no value given for event_measurement_name_tag value's tag key
  default_event_measurement = "dogstatsd_events"

Test Events

Telegraf conf file:

[[inputs.statsd]]
  ...
  parse_data_dog_tags = true
  parse_data_dog_events = true
  
  event_measurement_name_tag = "measurement_name"
  
  ## will default to this measurement name if value of above tagname
  ## is not provided
  default_event_measurement = "dogstatsd_events"
  ...

Event A

Input

The following two events were sent to the statsd listener:

echo -ne "_e{10,26}:Some title|Lorem ipsum dolor sit amet|t:info|#dc:us-east,city:miami\n_e{10,26}:Some title|Lorem ipsum dolor sit amet|#dc:us-west,city:san francisco" > /dev/udp/localhost/8125

Expected Output

Event stored in dogstatsd_events measurement (because the tag of measurement_name was not provided in event)

Actual Output (Passed)

Since config turned on to look for dogstatsd events, at collection interval verified in influxdb:

> select * from dogstatsd_events
name: dogstatsd_events
time                 alert_type city          dc      host                  msg_text                   msg_title  priority
----                 ---------- ----          --      ----                  --------                   ---------  --------
2019-01-17T20:02:49Z info       miami         us-east <redacted>            Lorem ipsum dolor sit amet Some title normal
2019-01-17T20:02:49Z info       san francisco us-west <redacted>            Lorem ipsum dolor sit amet Some title normal

Event B

Input

The following two events were sent to the statsd listener (note the extra tag key measurement_name, and event type and priority in the second event):

echo -ne "_e{10,26}:Some title|Lorem ipsum dolor sit amet|t:info|#dc:us-east,city:miami,measurement_name:dogstatsd_events_aux\n_e{10,26}:Some title|Lorem ipsum dolor sit amet|t:warning|p:low|#dc:us-west,city:san francisco,measurement_name:dogstatsd_events_aux" > /dev/udp/localhost/8125

Expected Output

Event stored in dogstatsd_events_aux measurement (because that's the value of the tag measurement_name, which config takes)

Actual Output (Passed)

Since config turned on to look for dogstatsd events, at collection interval verified in influxdb:

> select * from dogstatsd_events_aux
name: dogstatsd_events_aux
time                 alert_type city          dc      host                  measurement_name     msg_text                   msg_title  priority
----                 ---------- ----          --      ----                  ----------------     --------                   ---------  --------
2019-01-17T20:21:31Z info       miami         us-east <redacted>            dogstatsd_events_aux Lorem ipsum dolor sit amet Some title normal
2019-01-17T20:21:31Z warning    san francisco us-west <redacted>            dogstatsd_events_aux Lorem ipsum dolor sit amet Some title low

[danielnelson]

I don't think we will want to import the datadog parser as it brings in too many dependencies and overhead for Telegraf, in particular the metrics package.

I'd also like to trim down some of the new options. Allowing the converter processor to set the measurement name to a tag value could achieve the same functionality as event_measurement_name_tag. This has been discussion on a few other issues and should be simple to implement. The default_event_measurement could be by default the name of the plugin "statsd" as is used in other plugins, but could be changed using the existing name_override setting.

[schahal]

@danielnelson agreed, very valuable input and makes sense (pulling in the dd project seemed severely overkill), I'll try to incorporate these changes the next time I'm on the branch and testing.