diff --git a/src/how-to/administrate/cassandra.md b/src/how-to/administrate/cassandra.md index b425fc5..6143ae5 100644 --- a/src/how-to/administrate/cassandra.md +++ b/src/how-to/administrate/cassandra.md @@ -105,3 +105,144 @@ cqlsh> SELECT name FROM galley.team WHERE team = e93308fc-1676-4d53-af15-4b7f5fa7599a; ``` + +## Migration + +The following process was made in particular for a K8ssandra migration, but the same process can be applied in a cassandra-to-cassandra migration. + +> **⚠️ Important:** +> This migration involves approximately **1 hour of downtime**. +> Plan accordingly and notify your users. +> Shut down `wire-server` or block traffic so no more writes happen to Cassandra (in any manner you see fit or are comfortable with) before starting the migration. +> Have users create backups for extra safety. + +Tools used: +- `nodetool` +- `sstableloader` +- `cqlsh` + +Both tools come with our solutions for Cassandra and K8ssandra. + +### Flush data on all nodes + +Flush the data on each Cassandra node in the old cluster to ensure all in-memory writes are persisted to disk. + +```bash +nodetool flush brig +nodetool flush galley +nodetool flush spar +nodetool flush gundeck +``` +--- + +### Copy SSTables + +Copy SSTables from one Cassandra node for the required keyspaces (brig, spar, gundeck, and galley). + +If Cassandra was installed using Wire's Ansible playbook, data will be under `/mnt/cassandra/data`. + +```bash +mkdir backup +cp -r /mnt/cassandra/data/brig backup/ +cp -r /mnt/cassandra/data/spar backup/ +cp -r /mnt/cassandra/data/galley backup/ +cp -r /mnt/cassandra/data/gundeck backup/ +``` +--- + +### Prepare your new cassandra/k8ssandra cluster +Install and prepare your Cassandra/K8ssandra cluster. We recommend building your own cluster charts, but, you can also use our testing [solution](https://github.com/wireapp/wire-server-deploy/blob/master/offline/k8ssandra_setup.md) and charts if needed. + +> **Note:** Some modifications may be required for your environment if you decide to use our k8ssandra solution. + +--- + +### Update Service References +Change all service references to Cassandra hosts (`cassandra.host`) in `values/wire-server/values.yaml`. + +Example: +```yaml +cassandra-migrations: + # images: + # tag: some-tag (only override if you want a newer/different version than what is in the chart) + cassandra: + host: + replicationFactor: 3 +``` + +Or apply a `sed`: + +```bash +sed -i 's///g' values/wire-server/values.yaml +``` + +> **Note:** If you have used our K8ssandra charts, the name of the new service will be `k8ssandra-cluster-datacenter-1-service.database` + +--- + +### Reinstall wire-server + +Reinstall `wire-server` so migration jobs can apply the required keyspace and table structure to your new cluster. + +--- + +### Move backup into the new cluster + +For Cassandra just copy it to one of the nodes in the new cluster. +For K8ssandra copy files directly into one of the datacenter pods. + +> **Note:** You could also make a volume mount for the pods, but it will require modifying the charts as the support is not in there natively. + +Example using `kubectl cp` for K8ssandra: + +```bash +kubectl cp backup k8ssandra-cluster-datacenter-1-default-sts-0:/tmp -n +``` +--- + +### Run sstableloader + +Run `sstableloader` for each table in each keyspace. +For simplicity and automation, the following script can be used. +Adjust BACKUP_DIR and HOSTS for your setup. + +HOSTS can point to a single or multiple nodes. For K8ssandra migration it is recommended to point it to K8ssandra headless service. + +`sstableloader` will discover the cluster topology and stream data accordingly. + +```bash +#!/bin/bash +set -euo pipefail + +BACKUP_DIR="backup" + +HOSTS="" + +for keyspace in "$BACKUP_DIR"/*; do + [ -d "$keyspace" ] || continue + echo "===== Keyspace: $(basename "$keyspace") =====" + + for table in "$keyspace"/*; do + [ -d "$table" ] || continue + echo "----- Loading table: $(basename "$table") -----" + sstableloader -d "$HOSTS" "$table" + done +done + +``` + +#### Notes on duration +- tested on a small DB with 2000 fresh users (15-20 minutes) +- for production databases longer in use a longer migration is expected + +--- + +One the script has finished execution, the migration is complete. +You can verify by comparing the count of tables between the old and the new cluster using `cqlsh`, for example: + +```cqlsh +SELECT COUNT(*) from brig.user; +``` + +The count between the two should be the same. +Once verified, you can bring the `wire-server` service back online.