Scylla Monitoring Advisor Inbalanced Connections

ScyllaDB
,
1

Hi

We’re running Scylla in Kubernetes with gocq driver, I’ve replaced the cql driver with the scylla one and so far I don’t have any cql optimizations showing in monitoring.

Though I consistently see the connections are imbalanced and at times the SSTable is also showing a warning. I posted a couple of screenshots, I see the load at least on the charts is not that even at times, but the reads and writes seem pretty even.

image
image850×390 35.9 KB

image
image3438×418 146 KB

We have 3 nodes, with an RF of 3 and a CL or quorum. Each node has its own k8 service and we have a load balancer k8 service, which balances across the 3 nodes. This is the service I used to connect our client with and use 19042 port. I’m just trying to figure out why we have these warnings and if connecting with the individual k8 service for each node would be a fix? Or what else I can do to determine the source of the warnings?

2

Any more information needed on this?

3

What is the exact warning you are getting?
Also do you have any hot partitions?

4

Sorry for the late reply.

It’s more so the dashboard is constantly lit up with unbalanced connections

image
image1261×246 17.1 KB

No hot partitions, the only thing I notice and I assume this is a potential reason for the warnings now, though correct me if I’m wrong is:

wrong shard being assigned; please check that you are not behind a NAT or AddressTranslater which changes source ports; falling back to non-shard-aware port