Originally from the User Slack
must-gather.zip (9.2 MB)
@Sh4d1: Hello all! I’m hitting some performance issue on my scylla cluster (5 nodes on GKE with scylla operator, with 7 cores and 56GB reserved for each scylla node). You can see the p95/99 on the attached image. During peak load we are a bit below 60k write/s and 35k read/s. From what I read scylla should not be that slow with that number of requests. In the logs, I see quite a lot of
INFO 2025-05-14 12:31:53,719 [shard 5:stat] reader_concurrency_semaphore - (rate limiting dropped 2918 similar messages) Semaphore _read_concurrency_sem with 24/100 count and 2087042/158502748 memory resources: timed out, dumping permit diagnostics:
which iiuc, is that the cpu is overloaded, which makes sense with the view per shard (second image). My guess is that we’re using scylla in a bad way for some table:
• I have around 60 large cells (no more than 5M)
• 305 large partitions (some going over 10M), and CDC enabled on one of the table with the most large partitions
• a lot of different tables/keyspaces, more or less used.
Is that enough to show the slowness I’m seeing ? Or could it be something else too? Feel free to ask more questions if needed, and thanks in advance!
The spike on the read latency on the left was due to a huge spike in writes at that time*
Also, could this be linked I guess?
@avi: What kind of storage are you using?
@Maciej_Zimnoch: Have you followed https://operator.docs.scylladb.com/stable/architecture/tuning.html ?
Tuning | ScyllaDB Docs
btw Avi is disabling write back cache still a recommended thing to do on GCP Local SSDs?
@Sh4d1: Yep sorry, I’m using local nvme with raid0 (with write cache disabled on xfs), and the tuning should have been done by the operator (the jobs ran correctly)
And of course cpu sets are used (hence 7 and not 8, cause of k8s system component using some cpu)
@Maciej_Zimnoch: would you mind sharing a must-gather? I can double check if tuning was executed correctly
Gathering data with must-gather | ScyllaDB Docs
@Sh4d1: Yes, doing that, do you need the --all-resources ?
@Maciej_Zimnoch: no, default collects what we usually need
@Sh4d1: There you go!
@Maciej_Zimnoch: tuning looks ok. You’re using quite ancient Scylla version (5.4.x), I would recommend trying out the latest one
@Sh4d1: Yes the upgrade is planned somewhere on my backlog 
I guess it can always help indeed! Better to upgrade and then try to see if the issues are still there according to you?
@Maciej_Zimnoch: yes, newer versions contains lots of improvements
@Sh4d1: Makes sense 
will try to do it end of this week, will keep you updated here!
Sorry for the delay, I’m currently upgrading my cluster to 6.2.3. Should I see improvements directly?
Looks like the latency decreased quite a bit, still have some p99 above 50ms, with spikes up to 1-2s, but I guess it’s linked to our hot partitions?