# Abnormal cluster's node(s) behaviour. High CPU usage on 4/5 nodes **URL:** \ **Category:** ScyllaDB\ **Tags:** performance, unanswered, kubernetes\ **Created:** [June 6, 2025, 1:55pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885 "2025-06-06T13:55:03Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:55pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/1 "2025-06-06T13:55:03Z")
_Installation details_ ### ScyllaDB version: 6.2.2 ### Cluster size: 5 x 8vCPU/32GB RAM ### OS (RHEL/CentOS/Ubuntu/AWS AMI): ubuntu-jammy-22.04-amd64-server-20250327 Hi, we running ScyllaDB in k8s on dedicated EC2 instances with 5 nodes. On each node exists only scyllaDB-related pod and k8s service pods. After one month of usage increased CPU usage for only 4 nodes of 5 were detected, when 5th node use CPU for ~20-40%; while all others are in ~70% Details: - for this period workload not changed - all nodes receiving read/write requests with almost identical RPS - compaction are enabled on all nodes and based on history running in background as usual on ALL nodes - we do not have scylla manager - we do no do repair (in our case it is not needed at all) - we do not setup backup/restore - based on system.clients table all nodes has the same number of clients connected: ~70 - thombstone setup to cleanup on compaction (not on repair as its done by default) Is there a way to see what tasks ScyllaDB perform in the background? - I investigated via ‘nodetool tasks’ but did not found anything suspicious > Datacenter: eu-central-1 > ======================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > – Address Load Tokens Owns Host ID Rack > UN 100.65.105.82 500.14 GB 256 ? 99c64a19-f0eb-46c3-80c4-e290f7a5fd3e eu-central-1 > UN 100.65.61.129 544.89 GB 256 ? 51af2d90-587c-4951-819d-309c4ed1268e eu-central-1 > UN 100.66.85.110 531.18 GB 256 ? efe0d085-9a42-4fd0-abda-fa7bc3532752 eu-central-1 > UN 100.70.9.99 558.64 GB 256 ? 9cc55415-d641-450d-8b3e-d1c59c9eb047 eu-central-1 > UN 100.71.73.244 544.99 GB 256 ? c6a865b9-7030-4012-8c30-fb0addb22a0e eu-central-1 node **100.115.164.196** - is a node with small CPU usage ScyllaDB monitoring stack dashboards Overview: ![Screenshot 2025-06-06 at 16.37.44](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/13efdf05e0d75215d749403e9c122082c897a46f.png) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:56pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/2 "2025-06-06T13:56:42Z")
Load - comparation node where CPU ~20%(100.) vs all others. On last image graph appear not for all period because we did restart the node: ![Screenshot 2025-06-06 at 16.28.55](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/9/92b4bed82da226135bf45f0959adbe1102cfa40c.jpeg) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:57pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/3 "2025-06-06T13:57:22Z")
![Screenshot 2025-06-06 at 16.29.09](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/9/984a86479c9a733873b074100a8fe77d48c7b24a.jpeg) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:57pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/4 "2025-06-06T13:57:57Z")
![Screenshot 2025-06-06 at 16.29.19](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/6/6bee7a3399f6092cb1489cc79c682932020f19dd.jpeg) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:59pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/5 "2025-06-06T13:59:06Z")
![Screenshot 2025-06-06 at 16.33.13](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/9/9e786a787b53886f87f7453a2347dcdda81d6bd9.jpeg) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 1:59pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/6 "2025-06-06T13:59:36Z")
Detailed read: ![Screenshot 2025-06-06 at 16.46.17](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/f/f12bae1865706ae8e5737b19fc0e3ae9a3ed5270.png) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 2:00pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/7 "2025-06-06T14:00:10Z")
Detailed write: ![Screenshot 2025-06-06 at 16.47.09](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/47844871b8e71f253455587a5c6faa390f6ed6e7.jpeg) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 2:01pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/8 "2025-06-06T14:01:22Z")
k8s Node exporter metrics (random node vs node with small CPU usage): ![Screenshot 2025-06-06 at 16.17.34](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/5/53b54e5557cccb2012c0a7b625311493dcfbeae4.png) I want to believe that one node with small CPU consumption is NORMAL one and all others are stuck somewhere in the high intensive CPU task. Will provide any extra details if needed. ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 2:04pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/9 "2025-06-06T14:04:34Z")
k8s Node exporter metrics - ScyllaDB node with small CPU usage: ![Screenshot 2025-06-06 at 16.18.01](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/6/609f747d7ff64e47366e974caf8ceceaba89aa7d.png) ---
**Author:** ![Podhornyi](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/podhornyi/32/1731_2.png) [@Podhornyi](https://forum.scylladb.com/u/Podhornyi)\ **Post date:** [June 6, 2025, 8:25pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/10 "2025-06-06T20:25:13Z")
~20%(100.) = ~20%(100.115.164.196) its a note to first screenshot on post ---
**Author:** ![yashwant\_chandrakar](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/yashwant_chandrakar/32/1740_2.png) [@yashwant\_chandrakar](https://forum.scylladb.com/u/yashwant_chandrakar)\ **Post date:** [June 13, 2025, 10:13am UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/11 "2025-06-13T10:13:19Z")
We are facing similar sort of issue, can someone guide how to fix this ---
**Author:** ![Gabriel](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/gabriel/32/60_2.png) [@Gabriel](https://forum.scylladb.com/u/Gabriel)\ **Post date:** [June 13, 2025, 1:53pm UTC](https://forum.scylladb.com/t/abnormal-clusters-node-s-behaviour-high-cpu-usage-on-4-5-nodes/4885/12 "2025-06-13T13:53:34Z")
@mflendrich take a look