From 2b76071d56a603c86f048d00ea7e6ae3b9309224 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Vedran=20Ivankovi=C4=87?= <33936733+Veki301@users.noreply.github.com> Date: Thu, 14 Aug 2025 13:16:41 +0200 Subject: [PATCH 1/2] add documentation for cassandra migration --- src/how-to/administrate/cassandra.md | 139 +++++++++++++++++++++++++++ 1 file changed, 139 insertions(+) diff --git a/src/how-to/administrate/cassandra.md b/src/how-to/administrate/cassandra.md index 529a577..fd5c542 100644 --- a/src/how-to/administrate/cassandra.md +++ b/src/how-to/administrate/cassandra.md @@ -79,3 +79,142 @@ On each server one by one: 4. Start the cassandra daemon process: `systemctl start cassandra` 5. Wait for your cluster to be healthy again. 6. Do the same on the next server. + +## Migration + +The following process was made in particular for a K8ssandra migration, but the same process can be applied in a cassandra-to-cassandra migration. + +> **⚠️ Important:** +> This migration involves approximately **1 hour of downtime**. +> Plan accordingly and notify your users. +> Shut down the `wire-server` before starting the migration. +> Have users create backups for extra safety. + +Tools used: +- `nodetool` +- `sstableloader` +- `cqlsh` + +Both tools come with our solutions for Cassandra and K8ssandra. + +--- + +### Prepare your new cassandra/k8ssandra cluster +Install and prepare your Cassandra/K8ssandra cluster. +For K8ssandra you can use our testing [solution](https://github.com/wireapp/wire-server-deploy/blob/master/offline/k8ssandra_setup.md) and charts as needed. + +> **Note:** Some modifications may be required for your environment if you decide to use our k8ssandra solution. + +--- + +### Update Service References +Change all service references to Cassandra hosts (`cassandra.host`) in `values/wire-server/values.yaml`. + +Example: +```yaml +cassandra-migrations: + # images: + # tag: some-tag (only override if you want a newer/different version than what is in the chart) + cassandra: + host: + replicationFactor: 3 +``` + +Or apply a `sed`: + +```bash +sed -i 's///g' values/wire-server/values.yaml +``` + +--- + +### Reinstall wire-server + +Reinstall `wire-server` so migration jobs can apply the required keyspace and table structure to your new cluster. + +--- + +### Flush data on all nodes + +Flush the data on each Cassandra node in the old cluster to ensure all in-memory writes are persisted to disk. + +```bash +nodetool flush brig +nodetool flush galley +nodetool flush spar +nodetool flush gundeck +``` +--- + +### Copy SSTables + +Copy SSTables from one Cassandra node for the required keyspaces (brig, spar, gundeck, and galley). + +If Cassandra was installed using Wire's Ansible playbook, data will be under `/mnt/cassandra/data`. + +```bash +cp -r /mnt/cassandra/data/brig backup/ +cp -r /mnt/cassandra/data/spar backup/ +cp -r /mnt/cassandra/data/galley backup/ +cp -r /mnt/cassandra/data/gundeck backup/ +``` +--- + +### Move backup into the new cluster + +For Cassandra just copy it to the new cluster. +For K8ssandra, make a volume mount pointing to the backup directory (if supported by your charts), or copy files directly into one of the datacenter pods. + +Example using `kubectl cp` + +```bash +kubectl cp backup k8ssandra-cluster-datacenter-1-default-sts-0:/tmp -n +``` +--- + +### Run sstableloader + +Run `sstableloader` for each table in each keyspace. +For simplicity and automation, the following script can be used. +Adjust BACKUP_DIR and HOSTS for your setup. + +HOSTS can point to a single or multiple nodes. For K8ssandra migration it is recommended to point it to K8ssandra headless service. + +`sstableloader` will discover the cluster topology and stream data accordingly. + +```bash +#!/bin/bash +set -euo pipefail + +BACKUP_DIR="backup" + +HOSTS="" + +for keyspace in "$BACKUP_DIR"/*; do + [ -d "$keyspace" ] || continue + echo "===== Keyspace: $(basename "$keyspace") =====" + + for table in "$keyspace"/*; do + [ -d "$table" ] || continue + echo "----- Loading table: $(basename "$table") -----" + sstableloader -d "$HOSTS" "$table" + done +done + +``` + +#### Notes on duration +- tested on a small DB with 2000 fresh users (15-20 minutes) +- for production databases longer in use a longer migration is expected + +--- + +One the script has finished execution, the migration is complete. +You can verify by comparing the count of tables between the old and the new cluster using `cqlsh`, for example: + +```cqlsh +SELECT COUNT(*) from brig.user; +``` + +The count between the two should be the same. +Once verified, you can bring the `wire-server` service back online. \ No newline at end of file From d6da76848de396a56b78e6886b3af935bb3fcd7a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Vedran=20Ivankovi=C4=87?= <33936733+Veki301@users.noreply.github.com> Date: Wed, 10 Sep 2025 12:23:35 +0200 Subject: [PATCH 2/2] refactor cassandra migration document for clarity --- src/how-to/administrate/cassandra.md | 68 +++++++++++++++------------- 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/src/how-to/administrate/cassandra.md b/src/how-to/administrate/cassandra.md index fd5c542..f146e8a 100644 --- a/src/how-to/administrate/cassandra.md +++ b/src/how-to/administrate/cassandra.md @@ -87,7 +87,7 @@ The following process was made in particular for a K8ssandra migration, but the > **⚠️ Important:** > This migration involves approximately **1 hour of downtime**. > Plan accordingly and notify your users. -> Shut down the `wire-server` before starting the migration. +> Shut down `wire-server` or block traffic so no more writes happen to Cassandra (in any manner you see fit or are comfortable with) before starting the migration. > Have users create backups for extra safety. Tools used: @@ -97,11 +97,37 @@ Tools used: Both tools come with our solutions for Cassandra and K8ssandra. +### Flush data on all nodes + +Flush the data on each Cassandra node in the old cluster to ensure all in-memory writes are persisted to disk. + +```bash +nodetool flush brig +nodetool flush galley +nodetool flush spar +nodetool flush gundeck +``` +--- + +### Copy SSTables + +Copy SSTables from one Cassandra node for the required keyspaces (brig, spar, gundeck, and galley). + +If Cassandra was installed using Wire's Ansible playbook, data will be under `/mnt/cassandra/data`. + +```bash +mkdir backup +cp -r /mnt/cassandra/data/brig backup/ +cp -r /mnt/cassandra/data/spar backup/ +cp -r /mnt/cassandra/data/galley backup/ +cp -r /mnt/cassandra/data/gundeck backup/ +``` +--- + --- ### Prepare your new cassandra/k8ssandra cluster -Install and prepare your Cassandra/K8ssandra cluster. -For K8ssandra you can use our testing [solution](https://github.com/wireapp/wire-server-deploy/blob/master/offline/k8ssandra_setup.md) and charts as needed. +Install and prepare your Cassandra/K8ssandra cluster. We recommend building your own cluster charts, but, you can also use our testing [solution](https://github.com/wireapp/wire-server-deploy/blob/master/offline/k8ssandra_setup.md) and charts if needed. > **Note:** Some modifications may be required for your environment if you decide to use our k8ssandra solution. @@ -126,6 +152,8 @@ Or apply a `sed`: sed -i 's///g' values/wire-server/values.yaml ``` +> **Note:** If you have used our K8ssandra charts, the name of the new service will be `k8ssandra-cluster-datacenter-1-service.database` + --- ### Reinstall wire-server @@ -134,38 +162,14 @@ Reinstall `wire-server` so migration jobs can apply the required keyspace and ta --- -### Flush data on all nodes - -Flush the data on each Cassandra node in the old cluster to ensure all in-memory writes are persisted to disk. - -```bash -nodetool flush brig -nodetool flush galley -nodetool flush spar -nodetool flush gundeck -``` ---- - -### Copy SSTables - -Copy SSTables from one Cassandra node for the required keyspaces (brig, spar, gundeck, and galley). - -If Cassandra was installed using Wire's Ansible playbook, data will be under `/mnt/cassandra/data`. - -```bash -cp -r /mnt/cassandra/data/brig backup/ -cp -r /mnt/cassandra/data/spar backup/ -cp -r /mnt/cassandra/data/galley backup/ -cp -r /mnt/cassandra/data/gundeck backup/ -``` ---- - ### Move backup into the new cluster -For Cassandra just copy it to the new cluster. -For K8ssandra, make a volume mount pointing to the backup directory (if supported by your charts), or copy files directly into one of the datacenter pods. +For Cassandra just copy it to one of the nodes in the new cluster. +For K8ssandra copy files directly into one of the datacenter pods. + +> **Note:** You could also make a volume mount for the pods, but it will require modifying the charts as the support is not in there natively. -Example using `kubectl cp` +Example using `kubectl cp` for K8ssandra: ```bash kubectl cp backup k8ssandra-cluster-datacenter-1-default-sts-0:/tmp -n