The ScyllaDB team is pleased to announce the release of ScyllaDB Monitoring Stack 4.16.0
Related Links
The main change in this release is the migration to the Grafana 13 format and to the ScyllaDB data source plugin 0.7.0.
Version updates for ScyllaDB Monitoring Stack 4.16.0
-
Prometheus upgraded to version 3.14.0
-
Grafana upgraded to version 13.2.0
Dashboards updates
Detailed Dashboard
-
Add panels for new metrics that track CQL request latency and timestamp drift #2883
-
CQL request latency — the end-to-end server-side latency of a CQL request, measured in the transport (CQL protocol) layer: from the moment the server starts processing the request until the response is written back to the socket. This measurement wraps around the existing coordinator and replica latencies rather than replacing them, and adds the work the CQL layer does itself — decoding the frame, looking up the prepared statement, authorization, binding parameters, waiting for CPU on a busy shard, and serializing the result. Comparing it against the coordinator latency shows how much request time is spent outside the data path.
-
CQL client timestamp drift — the gap between the timestamp the client attached to the request and the server’s clock when the request actually arrived. It’s useful for spotting delays that happen before a request reaches the CQL server (network or client-side queuing), and for catching clock skew between clients and nodes, which matters because Scylla resolves conflicting writes by timestamp.
-
Alternator Dashboard
-
Add additional Alternator metrics to Alternator dashboard #2871
-
Blocked and shed requests - requests that were delayed because Alternator hit its memory limit, and requests that were dropped outright under overload. Both are early warning signs that the server is taking on more work than it can sustain.
-
HTTP-level errors - failures reading a request off the socket or writing the reply back, which point to client or network problems rather than query execution.
-
Per-table capacity units - read and write capacity units consumed per table, the same unit DynamoDB users size and bill their workloads in, making it easier to compare usage against DynamoDB expectations.
Note that as Alternator is more efficient, some of the operations (like delete and update) will take fewer WCUs on Alternator than on DynamoDB. The WCU and RCU are an upper limit, and in practice the same load on DynamoDB would take more units.
- Batch item count - the distribution of how many items each BatchGetItem/BatchWriteItem request carries, useful for correlating latency spikes with unusually large batches.
- Authentication and authorization failures - rejected requests split by cause: bad credentials versus valid credentials without permission. Useful for catching misconfigured clients and for spotting credential problems.
Keyspace dashboard
-
Index rows are now hidden when the keyspace has no secondary indexes. Previously, the index section was rendered as an empty, meaningless row. Index rows now repeat per index, and row titles carry a table/index prefix so it’s clear which object each row describes. #2880
-
Removed the global tablet heatmap and graph. Aggregating tablet counts across the whole cluster on a keyspace-scoped dashboard produced a view that didn’t mean much, so it’s gone. #2880
Breaking Changes
Update ScyllaDB plugin to 0.7.0 #2889
The ScyllaDB data source plugin allows you to read tables (specifically, system tables with information about connections and warnings about large cells, rows, and partitions). The new version adds a safety mechanism for use in unsafe environments.
By default, the data source will only connect to the specified cluster.
This means you now need to specify a host by default.
You can read more about connection scope here.
Grafana 13.x format
While the new Grafana 13.x format brings many potential benefits, it is not backward compatible. This means that if you are running your own Grafana server, you need to update it to the latest Grafana 13 release.
Also, if you are using the template mechanism, note the changes.
Grafana Renderer security token #2864
Starting from Grafana 13.x, Grafana and the Grafana renderer must use a security token to communicate.
If you are connecting to the renderer directly, you will need to use that token. For that, you will need to either supply the token yourself or let the start-all.sh script generate a token and use it.