Add cluster mode support

[qjsoq]

This pull request introduces a new, comprehensive Helm chart named valkey-cluster for deploying a clustered Valkey instance on Kubernetes.
This chart is designed to handle the complexities of a clustered setup by leveraging a StatefulSet and an intelligent initialization script.

What This PR Does:

• StatefulSet Deployment: Ensures stable network identifiers and ordered deployment, which is critical for a clustered database.
• Introduces Dynamic Per-Pod PVCs: Implements volumeClaimTemplates to automatically provision a unique PersistentVolumeClaim for each cluster node, ensuring data persistence and isolation.
• Automated Cluster Initialization: A new init-cluster.sh script (managed via cluster_config.yaml) is introduced. This script handles complex cluster bootstrapping, node discovery, and dynamic joining logic for new or replacement pods.
• Graceful Master Failover: Adds a default preStop lifecycle hook. This critical feature ensures that when a master pod is scheduled for termination (e.g., during a rolling update), it first triggers a controlled failover to one of its slaves before shutting down, minimizing downtime.
• Adds Headless Service: Creates the necessary headless service for stable network discovery between pods within the StatefulSet.
• Exposes Metrics Exporter: Adds a dedicated Kubernetes Service for the Prometheus metrics exporter sidecar. This makes cluster metrics easily discoverable and scrape-able for monitoring systems.
• Adds a PodDisruptionBudget to protect the cluster during voluntary disruptions.

How the Auto-Clustering Works
The core of the cluster logic resides in the init-cluster.sh script, which runs as a background process in the main container:

Peer Discovery: Each pod uses the Kubernetes headless service to discover its peers.

Joining an Existing Cluster: If a new pod starts and discovers an already healthy cluster, it uses the CLUSTER MEET command to join. It then intelligently finds a master with a deficit of replicas and assigns itself as a replica using CLUSTER REPLICATE.

Initial Cluster Creation: If no cluster is found, the first pod (-0) takes on the role of the initiator. It waits for all other pods to become ready and then bootstraps the entire cluster using the valkey-cli --cluster create command with the appropriate number of replicas.

Resiliency: The script can handle pods restarting by attempting to re-join the cluster and forget its old failed entry if necessary.

Here are some screenshots demonstrating the chart in action after a successful deployment.
For deployment these settings were set:

replicaCount: 6
metrics:
  enabled: true
dataStorage:
  enabled: true
  requestedSize: "8Gi"

1. Cluster Nodes Output

This shows the successful formation of a 6-node cluster (3 masters, 3 replicas).

2. Running Pods and PVCs

This confirms that the StatefulSet has successfully created all pods and their corresponding PersistentVolumeClaims.

3. Services

[gugu]

This PR also changes deployment to statefulset for non-cluster use cases. While it is not a compatible change I believe statefulset is more suitable for valkey workloads

[sgissi]

Thanks for submitting the PR. I would prefer a single chart to make it more consistent, but I understand that Valkey Cluster is quite different. I'll take a closer look on this chart this week.

[macropin]

We've been kicking the tyres on this PR, and it looks very promising. We'd love to see this PR accepted.

A few thoughts (not all related specifically to this PR):

• Reliability of failover with a preStop hook remains to be tested.
• Seems to work fine with v9.0.0
• There are about half a dozen sleep x commands in the init scripts, while these are sometimes unavoidable they are nonetheless a code smell
• valkey-cli seems rough, having to echo yes to the tool on cluster creation is awkward. It should not expect input when being run non-interactively
• Its disappointing that "cluster mode" does not support a single primary with multiple replicas

During testing I refactored this PR locally to configure with

cluster:
  # Number of primary nodes in the cluster
  primaries: 3
  # Number of replicas per primary node
  replicasPerPrimary: 1

however until cluster mode supports a more arbitrary configuration then I don't think there is any benefit to this separation.

[qjsoq]

@sgissi
I just wanted to follow up about any updates regarding the review.
Also, I’ve recently added a few fixes for the readiness probe and cluster scaling up. Let me know if you have any questions about this PR!

[sgissi]

Hi @qjsoq, currently we are working on Replication and Sentinel, with Cluster being the next target. Ideally, I would like to keep a single chart to keep consistency and avoid repeating work across the two charts. At this point, I don't know yet if that is feasible or not. I estimate it will be a few weeks still until we are ready to review Cluster mode.

[lyatanski]

Wouldn't it be slightly more optimal if there is podManagementPolicy: Parallel set in the spec? On initial installation this should improve cluster creation time.