[POC] Full cluster Replication using Segment Replication and Remote Store

[ankitkala]

Description

Full cluster Replication using Segment Replication and Remote Store

Issues Resolved

opensearch-project#7222

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL:
• CommitID: a4182e1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: null ❌
• URL:
• CommitID: 42d6136
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[abargnesi]

Hi @ankitkala. I stumbled on your work here after researching alternatives to the opensearch cross cluster replication. The cross-cluster-replication plugin establishes a follower index that replays translog changes from the leader index that are incompatible with the index lineage (i.e., both indices on generate 5 may not be equivalent). The segrep approach you present here would use same index lineage (same commit points and segment file set) which theoretically means failover and failback operations would not require full replication unless the follower has a newer generation.

If follower does have a newer generation than leader, it can do full replication sync into a new index, then do an index alias switch once it's consistent. That way the stale follower index could still be readable during full sync.

Do you know if this work is ongoing?

I see RFCs in core opensearch here:

• [POC] Cross Cluster Replication as a core component opensearch-project/OpenSearch#7222
• [RFC] Cross Cluster Replication as a Core Component opensearch-project/OpenSearch#2872

Also in cross-cluster-replication:

• [RFC] Full Cluster Replication with Failover & Failback support opensearch-project/cross-cluster-replication#549

But there have not been any progress in a while. Maybe opensearch has moved on to an alternative strategies?

Thanks in advance!

[ankitkala]

Hi @abargnesi , your assessment is correct. right now there isn't anyone actively working on it. Feel free to pick up this work if you're interested.