Hi all,
We are running 8 node cluster with scylladb 6.0 version. We could see below message from logs across all the nodes . But overall cpu(8 cores) usage was not much. And also we observed most of the times all 8 shards were 100% from the Load metrics(Scylla dashboard). Can someone help share your thoughts and how to fix it.
Log message
[shard 5:stmt] reader_concurrency_semaphore - (rate limiting dropped 584 similar messages) Semaphore user with 61/100 count and 11331744/166807470 memory resources: timed out, dumping permit diagnostics:
permits count memory table/operation/state
2 2 581K in_user_history.recent_recommended_items/data-query/active/need_cpu
1 0 0B in_user_history.recent_page_views/mutation-query/waiting_for_admission
27 0 0B in_user_history.recent_recommended_items/mutation-query/waiting_for_admission
132 0 0B in_user_history.recent_page_views/data-query/waiting_for_admission
228 0 0B in_user_history.recent_recommended_items/data-query/waiting_for_admission