Create and test an actionable disaster recovery plan

[gankoji]

Building on the investigation of DO's snapshotting tools, we need a formal, actionable disaster recovery (DR) plan. This plan should be documented and, most importantly, tested. This issue includes creating the plan and running a DR drill to ensure we can recover from a disaster scenario.

There are two different disaster scenarios to consider:

1. The obvious one: a part of (or all) of our data, database hosts, droplets, disks, and spaces/buckets disappear into thin air. How do we get back to up and running?

2. The more subtle one: the one guy who made it all work and knows where everything is gets injured/sick/hit by a bus/sick of our crap. Something breaks. How do we fix it?