(feat) External access for ValkeyCluster (externalAccess.type: None | LoadBalancer | Ingress; maybe Proxy in future)

[sandeepkunusoth]

Add an externalAccess configuration to ValkeyCluster to let users expose the cluster outside Kubernetes via LoadBalancer or Ingress(to share single loadbalancer between multiple valkey clusters). Default( None).

spec:
  externalAccess:                           # default: {}
    enabled: false                          # default: false
    type: LoadBalancer | Ingress     

[xtllli]

Proposal: Use Per-Pod NodePort Services for Valkey External Access

To enable client services in other K8s clusters to access Valkey clusters, we propose implementing a NodePort-based service discovery and connectivity model for Valkey on K8s.

Implementation Details:

The solution requires two types of NodePort services to enable direct client-to-pod communication, which Valkey requires:

1. Per-Pod NodePort Services (for Data and Cluster Traffic):
   • A separate NodePort service is created for each Valkey pod. Each NodePort service provides 2 port bindings: node_port->pod port 6379, and node_cluster_port->pod port 16379.
   • Valkey processes announce its address by:

     cluster-announce-ip: node_ip
     cluster-announce-port: node_port
     cluster-announce-bus-port: node_cluster_port
     

   • This unique address is what the Valkey pod advertises in its topology, ensuring that both peers (for gossip) and external clients (for direct data access) use the same address.
2. Cluster Discovery NodePort Service (for Initial Topology Lookup):
   • A single, cluster-level NodePort service is set up to match all Valkey pods.
   • Clients use a DNS record (for example, external-dns) that resolves to a list of Node IPs and the common Cluster Discovery Port (e.g., 36379) for an initial connection.
   • This initial connection is used to issue the CLUSTER SLOTS command, which returns the complete topology, including the per-pod NodeIP and NodePort for all cluster nodes, allowing the client to establish direct connections for all subsequent traffic.

Why NodePort is Preferred:

• Production-Verified: A similar method is already used in production for other stateful workloads, such as Kafka brokers (Strimzi) and TiDB.
• Cost and Complexity Avoidance: It avoids the high cost and operational complexity of setting up an NLB for every single Valkey pod, and removes the need for building a proxy layer that is 100% Redis client compatible. Ingress is also not ideal since redis connections run on L4 but ingress is a L7 load balancer.

Drawback:

• Performance: NodePort increases latency by extra network hops and potential bandwidth limitations for traffic going through the same kube proxy. This may be mitigated by directly hitting the node running the desired valkey pod, but cannot be completely addressed.
• Node ports exhaustion: there are at-most 10000s of available node ports for each k8s cluster and utilizing 2 per valkey pod can be a concern.
• Network accessibility: node_ip across k8s clusters need to be unique, resolvable, and accessible. It requires non-overlapping CIDRs and peered VPCs.

Please share your thoughts and any other better alternatives. Thanks!

[jdheyburn]

@ysqyang Has a POC PR on this: #76

I believe we would need to flesh out all the requirements so that we can incorporate a well designed CRD

[jdheyburn]

I started a discussion for this, as comments can then be threaded:

• [Idea] External access for Valkey #77

[sandeepkunusoth]

I have seen few places where node port is not recommended in production due to ports exposing on nodes. I think ingress also can be option just incase if anyone to use.

Just fyi, We are also working on setting up service mesh between k8s clusters so that pod ips become routable without any ingress or load balancer. for service mesh approach we dont have any requirements from operator for this to work

[jdheyburn]

I think this is why we need to discuss about ensuring we have a design that supports as many use cases as possible to avoid breaking changes in the future. While we probably don't need to support ALL cases, if we make the spec flexible enough to allow for something like a service mesh, that too is fine as well. But I am interested in hearing more about the service mesh requirement