Upgrading ElasticSearch for Scale

Upgrading the Cluster

There are multiple ways elastic.co provides to upgrade your elasticsearch cluster. This guide explains how to set up a separate cluster and transfer the data over. While this method requires additional hardware, it will also provide the least amount of downtime for Scale.

Moving from 2.4 to 5.6

A direct upgrade from 2.4 to 5.6 will usually require a full cluster restart, as described here. However, setting up a new cluster in parallel will ensure that Scale downtime is minimal.

This method requires the following:

• Minimum of 3 servers to start the new 5.6 cluster.
• Snapshot of current elasticsearch cluster stored in a shared directory or s3 bucket. Instructions on setting up snapshots are here.

Let's Get Started

Install elasticsearch 5.6 on each of your new clusters' nodes. Official documentation for installation is here. You will of course want to set up zen discovery and other important settings, but these can generally be copied from your 2.4 cluster nodes.

Migrating Data

Once your cluster is set up and working properly, you can add the existing snapshot repository the same way you added it here... with one exception: BE SURE TO ADD IT AS READONLY AT THIS POINT. Elasticsearch 5.6 can read and write to a snapshot repository created in 2.4 without issue, however once the 5.6 cluster "takes ownership" of the repository anything new from the 2.4 cluster will not be visible. You want to make sure everything is migrated successfully and working properly before then.

An example...

curl -XPUT http://<elasticurl>:9200/_snapshot/<repository_name> -d '{ "type": "fs", "settings": { "location": "/folder/subfolder", "compress": true, "readonly": true}}'

Now you can restore your latest backup. More detail on how to do that here. Depending on how much data you have, this could take minutes, hours or even days.

Once the restore is completed, depending on how long it took, you might want to create another snapshot on the 2.4 cluster for any indices that were created while the 5.6 cluster was restoring data. Ideally when you switch over there should be less than 24h of data missing in the 5.6 cluster.

Make the Switch!

Now you have two Elasticsearch clusters side by side, with 5.6 only missing data from anything run today.

Step one is to pause the Scale scheduler. This prevents any new data going into elasticsearch.

Now make one final snapshot on 2.4 of today's index, then restore that data into 5.6. (It shouldn't take very long for one index.)

Next step is to open DCOS, and go to Services->Scale->Configuration. Hit the edit button and update the ELASTICSEARCH_URL value to point to your new 5.6 cluster.

Review and run, and Scale will restart, passing along the new Elasticsearch URL to Fluentd along the way.

After Scale finished starting up you can re-enable the scheduler, and things will start moving normally again.

Finally, once you have verified everything is running properly, delete the snapshot repository from the 5.6 cluster (this only remove the reference from the cluster, and does not remove any snapshot data) and then re-add it without the readonly tag.

You can now take down your 2.4 cluster, or optionally wipe/install 5.6 on them and add them to your 5.6 cluster.