(feat) Deploy replicas on separate node / zones / regions

[RysoBeco]

The operator currently creates set of pods taking into consideration affinity, toleration and other standard k8s scheduling related features. Unfortunately there is no simple way to avoid deploying master and replica on same node / zone or region. It limits replica usability for many use cases.

Operator could check Valkey topology and assign role (shard & master/replica role) before pod scheduling and add custom filter to k8s scheduler.

Spec could look like this:

spec:
  shards: 3
  replicas: 2
  replicaSpreadConstraints:
    - topologyKey: zone
      maxReplicas: 2
    - topologyKey: node
      maxReplicas: 1

Diagram below illustrates how would valkey-operator schedule last pod created in the above example.

[jdheyburn]

This is definitely something we want, we'll look to include it after we have the MVP v0.1.0 released.

I am thinking that the operator deploys these with soft contraints initially (preferred affinity). Then if that is not enough, we can enable hard constraints (required affinity) from a user config.

[RysoBeco]

Sure, preferred affinity would be a useful feature.

Without k8s scheduler, it may be implemented building multiple deployments. For instance:

for i := 0; i < cluster.Spec.Shards; i++ {
  kind: Deployment
  metadata:
    name: cluster.Name + "-" + shardName[i]
    labels:
      valkey.io/shard: shardName[i]
  spec:
    replicas: cluster.Spec
    template:
      spec:
        affinity:
          podAntiAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                topologyKey: cluster.replicaSpreadConstraints.topologyKey
                labelSelector:
                  matchExpressions:
                  - key: "valkey.io/shard"
                    operator: In
                    values:
                    - shardName[i]
              weight: claster.replicaSpreadConstraints.weight
}

[jdheyburn]

@RysoBeco Thanks! We are looking to implement Singleton Deployments/StatefulSets to allow us the most control over pod placement.

• [Design] Pod management design #14