-
We have a test stand with stretched scylladb cluster:
- 10.0.0.2 and 10.0.0.5 in dc1
- 10.0.0.3 in dc2
- RF=3
- CL=2
-
the attacking host placed in dc1
- use gocql driver
- use policy DCAwareRoundRobinPolicy in gocql driver (so assumed we must read from nearest hosts)
-
All nodes are same
- scylla 4.5.3 in docker
- OS - Ubuntu 22.03
- 4 vcpu
- 16GB RAM
- SSD 160G
-
what is the reason that after the time expires, the application starts reading from host 10.0.0.3 (which is obviously located further - in another data center)
-
And after some time, the situation changes exactly the opposite.
- This is clearly seen in the attached image:
- What metrics to look out for?
- Or any other assumptions…
- This is clearly seen in the attached image:
- This night result:
- As can be seen from the graph: the first time reading was from the nearest nodes. But not for long
A few minutes later, a host located in another DC took the lead
When you have a multi-DC setup, make sure the Consistency Level you use for you reads is a LOCAL_ one: LOCAL_ONE or LOCAL_QUORUM. Otherwise, the replicas will be selected from among all the nodes in the cluster, disregarding DC.
See Consistency Levels | ScyllaDB Docs.