Originally from the User Slack
@Nick_Vladiceanu: Hey folks
we started to observe C++ exceptions in our cluster (Scylla version 5.2), resulting in errors on the clients. Where should we look first, and how can we find the C++ exceptions? Any help appreciated
The only unusual logs I find on some of the nodes is
INFO 2025-04-22 18:13:44,548 [shard 0] reader_concurrency_semaphore - (rate limiting dropped 8931 similar messages) Semaphore _read_concurrency_sem with 36/100 count and 1467312/146381209 memory resources: timed out, dumping permit diagnostics:
permits count memory table/description/state
11 11 496K system_schema.columns/shard-reader/inactive
14 14 470K system_schema.tables/shard-reader/inactive
8 8 360K system_schema.columns/shard-reader/active/used
3 3 102K system_schema.tables/shard-reader/active/used
2 0 5K system_schema.columns/multishard-mutation-query/active/unused
Thanks
@avi: 5.2 is long out of support, please upgrade to a supported version
@Nick_Vladiceanu: I guess this is a more generic question: where the C++ exceptions can be found? among regular logs?
@avi: Some C++ exceptions (e.g. timeouts) are swallowed by the code and converted into metrics (e.g. timeout metrics). The rest are in the logs.
@Nick_Vladiceanu: would they appear in the logs for the log level WARN, or require a lower log level? It was a bit difficult to understand what those C++ exceptions were about, no exceptions in the logs.
@avi: WARN or ERROR. They’re likely timeouts. Check the dedicated timeout metrics.