Investigate adding a built-in data backup solution

[dianabarsan]

Is your feature request related to a problem? Please describe.
We've had multiple production instances facing data loss due to disk failures, database corruption of unknown sources, where these instances either did not have backups set up or the backups were unusable.

Describe the solution you'd like
I'm proposing we investigate a built-in backup solution.
This should not be considered a full backup solution, but rather an opt-out setup where critical data is saved in a secondary location. Think first-aid-kit vs hospital or fire extinguisher vs fire brigade.

My initial thoughts would be to have a sidecar container that has access to two data volumes, one of them being the CouchDb data volume and the other being the secondary storage. Depending on the instance setup, this provides more or less safety.
This container would save the CouchDb `medic data shards to the separate location. This will exclude indexes and secondary databases (however, we would maybe choose to include the _users database as well). Saving would happen daily (? or be configurable through an env parameter for example).
The solution should also include scripts to restore data easily.
The solution should be e2e tested in our CI suite.

Describe alternatives you've considered
@jkuester suggested we somehow add the action to create the backups in cht-conf.
@mrjones-plip suggested to display a nudge to backup in the Admin app.
Other suggestions were to have an external service that does backup.
Considerations: backups may need to be sent to different vm’s, disks, data centers.

Happy to hear thoughts!

[jkuester]

Okay, this idea is not fully formed, but we talked about both a "sidecar" docker container OR a cht-conf action. Was thinking more today and remembered that we already publish a docker image for cht-conf.....

[dianabarsan]

I think I'm not seeing the advantage of this being a cht-conf action. I think cht-conf has a different use-case altogether, and it would seem like we're just bloating it with actions that the user can't actually use.
Use action A) if this runs on the server, and action B) C) D) if running on the developer's machine. It just feels confusing to me.

[mrjones-plip]

One of the hacks with a side car I was excited about was the ability to "backport" this to pre-CHT 5.2. I put backport in quotes as I'm envisioning CHT Core would be entirely unaware of the feature. The sidecar could be added to our existing compose file and reach right into the couch data volume and trivially copy it to separate, backup volume.

We could actually bake the feature into a upgrade service release as it can write to the compose files for both couch and core 🤔

Another thought I was that the sidecar could actually be two very slim containers:

• one container that reaches into aforementioned couchdb volume and then exposes the couchdb shards to be easily read (backup) or written (restored) to. I envision a rando port like 4000 that would leverage dead simple protocol like SCP, but could be smart with something like rsync (or even borg?!)). Of course encrypted on the wire and with strong authenticated.
• a second thin container would know the credentials and use the first containers exposed port & protocol to copy the files to a data volume.

We deploy them both in the same compose file as a happy path. A happy ++ path is that you deploy the second container on another VM and BAMO!! off VM backup! OR...a second data center!? All you need is access to the 4000 port and the credentials and you're good to backup and restore. There's no reason to not have multiples of the 2nd container running!

Finally, it seems like we only wanted to have the happy path backup the data shards and not the index shards - massive savings on space as we estimate 70-90% of a deployments data is taken up by the indexes. This, however, seems like an implementation detail where it would be trivial to offer a full backup (opt in, of course) which would play really nice with the second container to be configured slightly differently when space is not at such an optimum.

Admittedly

[dianabarsan]

on another VM

That doesn't work in Docker local, because it needs docker swarm to do cross-machine communication. And our experience with swarm is that it's unreliable when the main machine goes down (the swarm needs to be recreated).
Maybe all we need is a volume on a different VM and keep the container on the same machine?

I'm reading this article that provides a lot of options for a variety of docker network volumes

[mrjones-plip]

That doesn't work in Docker local, because it needs docker swarm to do cross-machine communication

Yup yup - agreed! However, that's the trick we can do with the slim "backup server" I have. It goes out of it's way to securely expose a way to backup the files both within the docker network, but just as easily outside the docker network.

I've done up a demo below that uses our production Core CouchDB compose file, but with two extra containers: the backup-server and the backup-client. Critically, the backup server has ports: "2222:22" declared, which expose a backup protocol externally (SSH in this case, but could be what ever we choose).

In the first half the demo there's a happy path that just works with no extra config from the user. But then in the second half of the demo, JUST the backup client is spun up on a fully separate machine and all it has is the private key and the address of the backup server. This scales nicely in that the backup client can decide how much to backup (just shards or shards + indexes?) as well as how much retention to have (having only one backup that happily backs up your corrupt shards won't help, where as having 7 days of backups will help immensely).

Critically, this solves the problem nicely for a docker based deployment, but to do fully "on another VM" when hosting in k8s would require more setup. This setup could be done with an SSH tunnel sidecar as we've done for superset or a bastion host as we've done for CHT Sync. Otherwise, I don't think we should let perfect get in the way of good and if we add this to the 2 sidecars to the helm charts and they just backup within the same infra, this would be a big win. It would be a lot easier to document how to expose a k8s backup server if it's up and running than if it doesn't exist at all 🤷

compose files referenced in POC

Full CouchDB compose file with 2 extra sidecars:

services:
  couchdb:
    image: public.ecr.aws/medic/cht-couchdb:5.1.0-beta.1
    volumes:
      - ${COUCHDB_DATA:-./srv}:/opt/couchdb/data
      - cht-credentials:/opt/couchdb/etc/local.d/
    environment:
      - "COUCHDB_USER=${COUCHDB_USER:-admin}"
      - "COUCHDB_PASSWORD=${COUCHDB_PASSWORD:?COUCHDB_PASSWORD must be set}"
      - "COUCHDB_SECRET=${COUCHDB_SECRET}"
      - "COUCHDB_UUID=${COUCHDB_UUID}"
      - "SVC_NAME=${SVC_NAME:-couchdb}"
      - "COUCHDB_LOG_LEVEL=${COUCHDB_LOG_LEVEL:-info}"
    restart: always
    depends_on:
      - nouveau
    logging:
      driver: "local"
      options:
        max-size: "${LOG_MAX_SIZE:-50m}"
        max-file: "${LOG_MAX_FILES:-20}"
    networks:
      cht-net:

  nouveau:
    image: public.ecr.aws/medic/cht-couchdb-nouveau:5.1.0-beta.1
    volumes:
      - ${COUCHDB_DATA:-./srv}/nouveau:/data/nouveau
    restart: always
    logging:
      driver: "local"
      options:
        max-size: "${LOG_MAX_SIZE:-50m}"
        max-file: "${LOG_MAX_FILES:-20}"
    networks:
      cht-net:

  backup-server:
    image: hermsi/alpine-sshd
    environment:
      - ROOT_PASSWORD="password"
      - ROOT_KEYPAIR_LOGIN_ENABLED=true
    volumes:
      - ${COUCHDB_DATA:-./srv}:/opt/couchdb-data
      - ./authorized_keys:/root/.ssh/authorized_keys
    restart: always
    ports: "2222:22"
    networks:
      cht-net:

  backup-client:
    image: alpine:3.23
    volumes:
      - cht-backup:/opt/backup/couchdb
      - ./ed25519:/root/.ssh/ed25519
    restart: always
    command:
      - /bin/sh
      - -c
      - |
        apk update
        apk add openssh rsync
        watch -n 2 "rsync -av -e 'ssh -i /root/.ssh/ed25519 -o StrictHostKeyChecking=no'  root@backup-server:/opt/couchdb-data/shards/ /opt/backup/couchdb/."
    networks:
      cht-net:
    depends_on:
      backup-server:
        condition: service_started
        restart: true

volumes:
  cht-credentials:
  cht-backup:

networks:
  cht-net:
    name: ${CHT_NETWORK:-cht-net}

#  ssh  root@172.17.0.1 -p 2222 -i /root/.ssh/ed25519  -o "IdentitiesOnly=yes"  -o StrictHostKeyChecking=no

Slim Just backup compose file

services:
  backup-client-los-dos:
    image: alpine:3.23
    volumes:
      - cht-backup-los-dos:/opt/backup/couchdb
      - ./ed25519:/root/.ssh/ed25519
    restart: always
    command:
      - /bin/sh
      - -c
      - |
        apk update
        apk add openssh rsync
        watch -n 2 "rsync -av -e 'ssh -p 2222 -i /root/.ssh/ed25519 -o StrictHostKeyChecking=no'  root@172.17.0.1:/opt/couchdb-data/shards/ /opt/backup/couchdb/."

volumes:
  cht-backup-los-dos:

NB - video has audio annotation which is muted by default. un-mute to hear the explanation!

dual.sidecar.poc.demo.mp4

[mrjones-plip]

I'm reading this article that provides a lot of options for a variety of docker network volumes

Oh - fun! That looks like it could nicely compliment whichever solution we come up with! I think going with a backup feature that is an opt out solution that has a very happy path (as in zero config by the user) would be key though. Some of these more fancy volume features in the post could be a relatively simple add on so that wherever we backup to is more resilient than a basic docker volume.