What could be the reason for system_auth:roles to become top hot partition?
READS Sampler:
Cardinality: ~256 (256 capacity)
Top 10 partitions:
Partition Count +/-
(system_auth:roles) user1 19119 85
Is it possible that you create lots of connections instead of reusing them?
When we reach to this count ~19119 with system_auth:roles table, cql connection timeouts. How can we identify the maximum no of connections a cluster can support without timeouts?
You should be able to observe it in monitoring under “Scylla CQL” dashboard we have " Client CQL new connections by Instance" and “Client CQL connections by Instance”.
It’s more about new connection rate than current connection pool, although both have their limits depending on your hardware.
I dont see this panel in “Scylla SQL” dashboard. What metrics it checks?
Perhaps you don’t have the latest version? It checks scylla_transport_current_connections metric (and it’s rate).
Thank you, Added the panel to dashboad. Rate was less than 0 and at some point it goes to 3.7. How to interpret this?
I’d say it’s rather low value, by default it would mean connections per second. And how many connections do you have? (remember to make sure it’s aggregated over all nodes and shards)