Reader_concurrency_semaphore

Database Community
1

Hi all,
We are running 8 node cluster with scylladb 6.0 version. We could see below message from logs across all the nodes . But overall cpu(8 cores) usage was not much. And also we observed most of the times all 8 shards were 100% from the Load metrics(Scylla dashboard). Can someone help share your thoughts and how to fix it.

Log message
[shard 5:stmt] reader_concurrency_semaphore - (rate limiting dropped 584 similar messages) Semaphore user with 61/100 count and 11331744/166807470 memory resources: timed out, dumping permit diagnostics:
permits count memory table/operation/state
2 2 581K in_user_history.recent_recommended_items/data-query/active/need_cpu
1 0 0B in_user_history.recent_page_views/mutation-query/waiting_for_admission
27 0 0B in_user_history.recent_recommended_items/mutation-query/waiting_for_admission
132 0 0B in_user_history.recent_page_views/data-query/waiting_for_admission
228 0 0B in_user_history.recent_recommended_items/data-query/waiting_for_admission

2

Looks like you overloaded ScyllaDB, at least when it comes to CPU. Try adding more nodes to the cluster, or reducing the load.

Also, the report you posted seems incomplete. I says there are 61 count resources, but I only see 2 in the table below. Either parts are missing, or the table is from the a different report?