Server Updates Spring 2025

[cycomachead]

This issue is largely based on #147 from years ago...

High Level Plan

• backup the staging server
• create a new large DO droplet based on the staging server backup.
• do all the work to setup the new server
• put the primary server in maintenance mode.
• ensure the new server is in maintenance mode
• "promote" the new server to the primary IP
• wait a few minutes for the load balancer to be OK
• turn off maintenance mode AND the old server

Days later, delete the old server.

[cycomachead]

TODO: This document should be updated to include specific commands to run / links to DO pages, etc.

---

Detailed Migration Notes

0. Remaining Prep on the staging server

• verify nginx-amplify
• verify newrelic
• setup snapcloud_production db
• setup metabase access to the staging db
• figure out trellix EDR?
• clear all bash and pg history [incase passwords are there]
• verify if it is still impossible to mount a filesystem on two devices at once...
• compare firewall settings on both servers

1. Prep the new primary server

• when to copy production lets-encrypt data? copy cert files?
• update the device hostname?
• manually update .env in the repo

2. Maintenance Mode and Migration

• Turn on Maintenance Mode
• ensure nothing can be updated
• Click snapshot in DO on BOTH the storage volume and server
• Dump the db
• Copy the db to the new server
• Unmount the project storage volume
• remount the project storage value
• copy .env and run a diff

3. Promoting the new server

• verify the IP addr responds to queries
• swap in DO
• turn off maintenance mode on the new server
• turn off the old server
• wait a week, then delete the old server.