# High read latency

**URL:** <https://forum.scylladb.com/t/high-read-latency/2792>\
**Category:** ScyllaDB\
**Tags:** performance, operator, latency\
**Created:** [August 27, 2024, 11:53am UTC](https://forum.scylladb.com/t/high-read-latency/2792 "2024-08-27T11:53:37Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kobejn](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/kobejn/32/1125_2.png) [@kobejn](https://forum.scylladb.com/u/kobejn)\
**Post date:** [August 27, 2024, 11:53am UTC](https://forum.scylladb.com/t/high-read-latency/2792/1 "2024-08-27T11:53:37Z")

</div>

Hi,

I’ve scylla cluster deployed on self hosted kubernetes cluster with k8s operator.  
I have 3 nodes of 16 CPUs and 32GB of mem. Each node is using node local storage with local-path-provisioneron ssd.

When client service is started read latency reaches 4s at peak and hangs at 1-2s on average. That is with reads at 6kosp/s and that does not seem like a lot. I’ve found this in the logs:

> INFO 2024-08-26 14:10:20,132 [shard 8:stat] reader\_concurrency\_semaphore - (rate limiting dropped 1 similar messages) Semaphore \_read\_concurrency\_sem with 100/100 count and 2280960/38755368 memory resources: timed out, dumping permit diagnostics:  
> permits count memory table/operation/state  
> 61 61 1299K user\_activity.user\_internal\_ids/data-query/active/await  
> 39 39 928K user\_activity.device\_internal\_ids/data-query/active/await  
> 1 0 0B user\_activity.device\_internal\_ids/mutation-query/waiting\_for\_admission  
> 132 0 0B user\_activity.device\_internal\_ids/data-query/waiting\_for\_admission  
> 151 0 0B user\_activity.user\_internal\_ids/data-query/waiting\_for\_admission
> 
> 384 100 2228K total
> 
> Stats:  
> permit\_based\_evictions: 15  
> time\_based\_evictions: 0  
> inactive\_reads: 0  
> total\_successful\_reads: 298208  
> total\_failed\_reads: 2729  
> total\_reads\_shed\_due\_to\_overload: 0  
> total\_reads\_killed\_due\_to\_kill\_limit: 0  
> reads\_admitted: 299593  
> reads\_enqueued\_for\_admission: 67237  
> reads\_enqueued\_for\_memory: 0  
> reads\_admitted\_immediately: 234191  
> reads\_queued\_because\_ready\_list: 29875  
> reads\_queued\_because\_need\_cpu\_permits: 7514  
> reads\_queued\_because\_memory\_resources: 29848  
> reads\_queued\_because\_count\_resources: 0  
> reads\_queued\_with\_eviction: 4  
> total\_permits: 301438  
> current\_permits: 384  
> need\_cpu\_permits: 100  
> awaits\_permits: 100  
> disk\_reads: 100  
> sstables\_read: 115

I’m not sure how to interpret that, especially “100/100 count”, cpu usage is low, memory is not saturated, I want to blame IO but can’t find any metrics to support it.

Thank you

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/f/f4abd15436b0193b75c3b246fe44314254e7b789.png)

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [August 29, 2024, 4:28am UTC](https://forum.scylladb.com/t/high-read-latency/2792/2 "2024-08-29T04:28:17Z")

</div>

32GB of RAM for 16 CPUs is very little. ScyllaDB will use most of that, leaving very little for cache, which will make most of your reads go to disk. ScyllaDB has a hard-limit on the maximum amount of concurrent disk reads: 100. Once you have this many disk reads, new read request will be queued. This is what we see above. Some read requests sit in this queue for so long, waiting for their turn, that they time out while waiting.

We usually provide around 8GB of RAM per CPU. That is a much more comfortable amount, which leaves room for a healthy amount of cache, which improves latencies a lot.

---

<div class="post-metadata">

**Author:** ![kobejn](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/kobejn/32/1125_2.png) [@kobejn](https://forum.scylladb.com/u/kobejn)\
**Post date:** [August 29, 2024, 11:32am UTC](https://forum.scylladb.com/t/high-read-latency/2792/3 "2024-08-29T11:32:21Z")

</div>

Thank you, I’m starting to understand some aspects better like the reads limit and queue.

That was my first thought that there is not enough resources for scylla to run stable, but when I check metrics in grafana scylla uses only 8 CPUs and 25 GB of memory. As a new scylla user it’s hard for me to know what performance should I be expecting from scylla in terms of reads/writes per second depending on what resources I assign to scylla pods.

I know it depends on the actual queries but is there like a rule of thumb like each 2 CPUs and 16 GB or ram should result in at least 1000 reads per second so I know at least it I’m getting performance in order of values I should be.