Feature/watchdog

[renereimann]

This dragonfly script is intendet to be an alarm system for your setup. There are two different things that the script checks.

1. it checks if all your docker containers are running and do not have an error
2. you can do specific checks on endpoints to check for critical behaviour
   If a problem is detected, the script will send a message to slack to warn you.
   The script is run in a docker container and an example docker-compose.yaml file is added to see how to setup the docker container.
   The script is configured via a yaml config file. The syntax is similar to our normal dripline config files. An example yaml config file is included in the pull request. The system works fine in the Mainz setup.

[wcpettus]

Working from high-level comments to lower:

• This mixes two functionalities, one of which (docker container checking) makes a lot of sense to exist outside of the normal dripline ecosystem, the other less so. Maybe unifying the watchdog functionality in one place is the right strategy, but more discussion on this point might be useful.
  • As an example in DL2 there was a sensor_monitor which could check endpoint values for anomalous readings. This was more passive in that it watched the alerts exchange (and so wasn't actively querying). But with heartbeats and the docker watchdog the active query maybe isn't necessary for endpoints?
• The dripline authentication should probably be managed by scarab and draw from the authentications file, and not require specifying again here in the config file
• Do the changes to dripline:service send impact the implementation here (@nsoblath)?
• Robert was working on a slack relay service that lives within dripline. Without touching the value of having the watchdog exist totally distinct from the rest of the mesh, down the road we might want to unify the slack connection pieces.
  • Do you have thoughts on the relative merit of webhook (used here) vs token (used by Robert) for authentication/connection?

[renereimann]

Thanks a lot Walter,

• I agree that one can separate the endpoint checking from the docker container checking. I do it here for my on simplicity but I definitively can make a seperate script for both of them and run them in separate containers.
• I recently saw that SlowDash should also get a end-point checking feature so some coordination here would be nice.
• About token vs webhook: a webhook is bound to a specific channel and can only post messages. The verification is part of the URL. With a token you can use the chat.postMessage() function and you can post to several channels and you can respond to user input. So you would need that if you have some interactive multi-channel application / bot. Regarding security I think since webhook only has very limited permissions it is fine to use but I do not have more details on that.
• The changes to dripline:service send does not impact this application (as far as I can tell) since its running fine before and after the fix
• I guess there are smarter ways for the authentification however I do not yet understand scarabs purpose and abilities (in terms of authentication and also more in general).