-
Notifications
You must be signed in to change notification settings - Fork 966
Description
The problem/use-case that the feature addresses
We are running/planning to run Valkey in cluster mode on Kubernetes with external traffic being ingested via NodePort
services. Thus, the announced ports for a Valkey cluster node should be its assigned node port. Meanwhile, the cluster internally—both for the cluster bus and replication—should ideally use direct pod-to-pod communication with whatever port the NodePort
service(s) map to. This should apply to mixed plaintext/TLS ports as well as TLS-only operation.
Currently, the cluster-announce-port
and cluster-announce-tls-port
options exist to change which ports are exposed "externally" for a node, but there is no distinction between "end user" client connections and intra-cluster ones. So if the announced port is set to the exposed Kubernetes node port, replication will fail as replicas try to connect to their primary via the pod IP but a port it doesn't listen to.
Since clients generally don't need to access the cluster bus, cluster-announce-bus-port
doesn't necessarily need the same attention.
Description of the feature
Akin to #337 and its implementation to #736, add two (or three) new options to complement cluster-announce-client-ipv4
/cluster-announce-client-ipv6
:
cluster-announce-client-port
: client-visible version ofcluster-announce-port
cluster-announce-client-tls-port
: client-visible version ofcluster-announce-tls-port
- Maybe
cluster-announce-client-bus-port
: client-visible version ofcluster-announce-bus-port
These will be used in place of the default announced ports in response to CLUSTER NODES
commands, MOVED
responses etc., while the default ones continue to be used for cluster-internal operations.
Alternatives you've considered
One alternative we considered was not using TLS for replication, so the cluster-announce-port
can be internal-only while the TLS port is used for external communication. We'd like to enable TLS for everything though to also make use of usernames derived from client certificates as implemented in #1920 and completely disable password-based authentication.
Another alternative would be to have each cluster node pod listen on its assigned Kubernetes node port so they match up "coincidentally" with the announced ports. This can work as a stopgap for us, but might not be viable in all setups in general.