Hi , I’m running a 3-node ScyllaDB cluster, with each node provisioned with 8 vCPUs and 32 GB of memory.
Scylla service is down with below logs-
ay 2 17:57:44 vm-wynk-prd-01 scylla[1684]: Reactor stalled for 33 ms on shard 3. Backtrace: 0x4d768f2 0x4d75550 0x4d76800 0x7f3fea8c0a1f 0x1dccc73 0x2129437 0x21414ad 0x272d0fa 0x276b54d 0x276543d 0x272f5f2 0x1dfe7de 0x4d860e4 0x4d874c7 0x4da6515 0x4d59ffa 0x92a4 0x100322
May 2 17:57:44 vm-wynk-prd-01 scylla[1684]: Reactor stalled for 33 ms on shard 5. Backtrace: 0x4d768f2 0x4d75550 0x4d76800 0x7f3fea8c0a1f 0x2129480 0x21414ad 0x272d0fa 0x276b54d 0x276543d 0x272f5f2 0x1dfe7de 0x4d860e4 0x4d874c7 0x4da6515 0x4d59ffa 0x92a4 0x100322
May 2 17:57:44 vm-wynk-prd-01 scylla[1684]: Reactor stalled for 33 ms on shard 4. Backtrace: 0x4d768f2 0x4d75550 0x4d76800 0x7f3fea8c0a1f 0x4d47e50 0x4d4bc8d 0x2202af0 0x21294d0 0x21414ad 0x272d0fa 0x276b54d 0x276543d 0x272f5f2 0x1dfe7de 0x4d860e4 0x4d874c7 0x4da6515 0x4d59ffa 0x92a4 0x100322
May 2 17:57:44 vm-wynk-prd-01 scylla[1684]: Reactor stalled for 33 ms on shard 2. Backtrace: 0x4d768f2 0x4d75550 0x4d76800 0x7f3fea8c0a1f 0x21294c8 0x21414ad 0x272d0fa 0x276b54d 0x276543d 0x272f5f2 0x1dfe7de 0x4d860e4 0x4d874c7 0x4da6515 0x4d59ffa 0x92a4 0x100322
May 2 17:58:30 vm-wynk-prd-01 systemd[1]: Starting GCE Workload Certificate refresh…
May 2 17:58:30 vm-wynk-prd-01 gce_workload_cert_refresh[2192]: 2025/05/02 17:58:30: Done
May 2 17:58:30 vm-wynk-prd-01 systemd[1]: gce-workload-cert-refresh.service: Succeeded.
May 2 17:58:30 vm-wynk-prd-01 systemd[1]: Started GCE Workload Certificate refresh.
May 2 17:58:44 vm-wynk-prd-01 scylla[1684]: [shard 1] BatchStatement - Batch modifying 206 partitions in iptv.iptv_users_error_events is of size 715358 bytes, exceeding specified WARN threshold of 204800 by 510558.
May 2 18:00:42 vm-wynk-prd-iptv-scylla-01 sshd[2221]: rexec line 124: Deprecated option UsePrivilegeSeparation
May 2 18:00:52 vm-wynk-prd-iptv-scylla-01 scylla[1684]: [shard 4] BatchStatement - Batch modifying 249 partitions in iptv.iptv_users_error_events is of size 711383 bytes, exceeding specified WARN threshold of 204800 by 506583.
May 2 18:02:24 vm-wynk-prd-01 scylla[1684]: [shard 1] BatchStatement - Batch modifying 226 partitions in iptv.iptv_users_error_events is of size 714514 bytes, exceeding specified WARN threshold of 204800 by 509714.
scylla --version
5.1.19-0.20231127.0f2269afbf9c