Multi Data Centers Cluster on K8s + IPVS Loadbalancer environment

ScyllaDB
, ,
1

Installation details
#ScyllaDB version: 2025.2
#Cluster size: 2 k8s cluster with 5 nodes per each
os (RHEL/CentOS/Ubuntu/AWS AMI):

I want to configure SycllaDB multi-data center clustering between two k8s clusters.

Our company doesn’t use a public cloud, but rather our own IDC.
The k8s LoadBalancer is configured via IPVS, so from the pod’s perspective, the destination IP is the LoadBalancer IP.

Therefore, if I bind the listen-address to the pod IP, the connection fails due to RST packets.

However, if I bind the listen-address to the LoadBalancer IP,

the source IP of outgoing packets from the pod is the listen-address (LoadBalancer IP), so the connection fails to establish properly.

Since the two k8s clusters don’t share a common network, access is only possible through the LoadBalancer.

How can I configure multi-data center clustering?

2

Your IPVS LB appears to use DSR mode. Packets reach the pod with the destination IP = VIP, which the pod doesn’t own, so the kernel rejects the SYN and you see RST. Binding Scylla to the VIP “fixes” the inbound path but breaks outbound (the pod now sources packets from the VIP). That’s expected with DSR and won’t work for Scylla pods.

You need the LB to perform NAT or L4 proxying so that packets arriving to the pod have destination = podIP (or NodePort DNAT). Then you can keep listen_address on the pod and advertise a different broadcast address for cross DC peers. This is exactly what the Operator broadcast options are for.

Please note that exposeOptions are immutable, they cannot be changed after a ScyllaDB cluster is created.

In general, what you need to do is the following:

  • Make sure your LB delivers packets with DNAT/L4-proxy (not DSR).
  • In case you have Firewall / ACLs between the two k8s clusters, you need to make sure to open the relevant ports per pod VIP (TCP 7000, 7001, 9042, 9142).
  • Provision DC1 and wait until all pods are UN (kubectl -n scylla get pods, nodetool status).
  • Collect the VIPs from DC1 node Services.
  • Apply DC2 with those VIPs in externalSeeds. ( ScyllaCluster (scylla.scylladb.com/v1) | ScyllaDB Docs ).
  • Validate using nodetool status that it shows both DCs, and all nodes in UN.
  • Validate with SELECT peer, native_address FROM system.peers_v2; that you see the VIPs and not PodIPs.
1 Like