Add incident notifications guide for Slack and PagerDuty (GTM-2216)

[czepluch]

No description provided.

[linear]

GTM-2216 docs: slack and pagerduty alerts integration

[jacobdcastro]

Looks great. One nice addition could be describing our dedup logic (which we are still finalizing) for notifications. So users know/trust they won't be spammed with alerts.

[odyslam]

@jacobdcastro makes a good point, great catch.

@czepluch The context for that is found here: https://www.notion.so/phylax/Incidents-Debugging-2c085d07a17b809a8721ef94fc87907c?source=copy_link#2c085d07a17b80cab8cafc30d80fd015

Although the exact dedup logic could change to handle some very edge cases, the main notion will stay the same:

1. An incident is defined as the invalidation of an assertion over a time period of one hour
2. An incident can have many invalidating transactions
3. We do that because it's possible that the attacker manages to craft multiple different transactions that invalidate the same assertion, so logically they are the same incident.
4. We have a time period of one hour because that's a good first ballpark of how long we expect an attack to take
5. if an assertion is invalidated again after an hour, there is a high chance that it's a different attack and thus a different incident
6. This is a first approach and we expect to iterate based on user feedback

This is generic about all incidents and not specific to the alerts. The alerts are just a way that an incident is communicated to the end-user so they can go to the dashboard and root cause

Otherwise, the page looks great and I love the GIF.