# Compaction Storm slows down Scylla

**URL:** <https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958>\
**Category:** ScyllaDB\
**Tags:** troubleshooting, compaction\
**Created:** [September 25, 2024, 3:07pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958 "2024-09-25T15:07:48Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Womsel](https://avatars.discourse-cdn.com/v4/letter/w/47e85d/32.png) [@Womsel](https://forum.scylladb.com/u/Womsel)\
**Post date:** [September 25, 2024, 3:07pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/1 "2024-09-25T15:07:48Z")

</div>

Today we observed a weird behavior of ScyllaDB:

All of a sudden, the cluster became noticeable slow. Logs were not very interesting, but showed alot of compactions going on on some nodes of the cluster. Looking into the metrics in our prometheus/grafana, it seems to show that more and more nodes suffered from these intense compactions (going at IOPS-Limit of the underlying disks).

A little excerpt from the Logs (going on like this for hours):

```auto
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectentries4 09d022b0-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0mi6o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_5oots2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_5v4b42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0ffk02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_5t6v42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_31yc02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3ntdc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_4n6io2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_5mjo02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_4pje82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3edv42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0o7ww2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0902o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3jbc02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0556o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_360xs2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0k5b42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_15smo2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_13fr42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0di402rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3gj0w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3c8pc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_49w4g2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_02sb42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_4dyq82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_07acg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_45e342rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_5c9ao2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_3u14w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_2xo0g2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_21ils2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw8_0tktd2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectchangepointers 0930d570-7afb-11ef-b677-bb2e6950de47] Compacted 8 sstables to [/var/lib/scylla/data/pc/businessobjectchangepointers-0e6da7e0757d11e98f29000000000000/me-3gju_0dw8_0tktc2rw6a1kw0kvuv-big-Data.db:level=0]. 3MB to 3MB (~98% of original) in 1056ms = 3MB/s. ~1280 total partitions merged to 310.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectentries4 09d022b0-7afb-11ef-b677-bb2e6950de47] Compacted 32 sstables to [/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1381d2rw6a1kw0kvuv-big-Data.db:level=0]. 217kB to 41kB (~18% of original) in 110ms = 1MB/s. ~4096 total partitions merged to 1.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.customworkerstates 09e50a40-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw8_57yz42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw9_17icw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw8_3s3ow2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw8_2aqe82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw9_19fsw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw8_2tt4g2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw9_10v5s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw7_5tu0k2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.tntevents2 09e50a41-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw7_1tsts2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw8_55eds2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw7_1rvds2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw8_539802rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw7_1pq802rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw6_27ink2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw8_514282rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw7_21y1c2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw8_5a44w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw6_4s3zk2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw8_4tea82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw4_3f8q82rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.customworkerstates 09e50a40-7afb-11ef-b677-bb2e6950de47] Compacted 8 sstables to [/var/lib/scylla/data/pc/customworkerstates-92b8e11e884207c98f26000000000000/me-3gju_0dw9_1wl4w2rw6a1kw0kvuv-big-Data.db:level=0]. 450kB to 341kB (~75% of original) in 147ms = 3MB/s. ~1920 total partitions merged to 916.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.tntindex2 0a0fc3c0-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_47yog2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_3ab9c2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_2ndn42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_5x9gw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_368nk2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_4z6m82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_3z6bk2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_0qksg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_2l8hc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_413rk2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_3m3n42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_0i05c2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_5gjm82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_2ivls2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_5em682rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw7_2giq82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_1l0gw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_31qm82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw7_3zlr42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_2cnu82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_17xsg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_1r0io2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_4fw682rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_1xg002rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_3863k2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw7_5c1kw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw7_125gg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_4x9682rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw7_0c7tc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_2f0ps2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw6_1nl282rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw8_2uvpc2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.tntindex2 0a0fc3c0-7afb-11ef-b677-bb2e6950de47] Compacted 32 sstables to [/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_3ktcg2rw6a1kw0kvuv-big-Data.db:level=0]. 1MB to 147kB (~8% of original) in 95ms = 18MB/s. ~65536 total partitions merged to 153.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.tntevents2 09e50a41-7afb-11ef-b677-bb2e6950de47] Compacted 12 sstables to [/var/lib/scylla/data/pc/tntevents2-98f63fb0a41011e9b8d6000000000000/me-3gju_0dw9_1wsuo2rw6a1kw0kvuv-big-Data.db:level=0]. 7MB to 6MB (~99% of original) in 391ms = 17MB/s. ~1536 total partitions merged to 68.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectentries4 0a254790-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0yxps2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0uv402rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_0wsk02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_15d742rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_2hdlc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_3c8pc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_268cw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_38t8w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1espc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_34b7k2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1yav42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1nl282rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_3mqsg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1l86o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_2cvk02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_1381d2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw7_5tu0i2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.serialvaultids2 091907b0-7afb-11ef-b677-bb2e6950de47] Compacted 8 sstables to [/var/lib/scylla/data/pc/serialvaultids2-efe8f8923b7e589fcd00000000000000/me-3gju_0dw7_5tu0j2rw6a1kw0kvuv-big-Data.db:level=0]. 15MB to 15MB (~99% of original) in 1823ms = 8MB/s. ~8832 total partitions merged to 7835.
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.usagecounters2 0a2fa7d0-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw8_2r8j42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_0x09s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_0inao2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_1rno02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw5_1nss02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw4_3fgg02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw4_2ug9s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_5abuo2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_3mj2o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_4o93k2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw3_509742rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw6_3gqqo2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw3_4jjcg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw5_23g1s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw4_5x1r42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw5_2dipc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw5_1zl5s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw8_3q68w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw9_2k5wg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw5_3ovy82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw3_47j8w2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:09 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectentries4 0a254790-7afb-11ef-b677-bb2e6950de47] Compacted 17 sstables to [/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4elvk2rw6a1kw0kvuv-big-Data.db:level=0]. 5MB to 5MB (~97% of original) in 79ms = 67MB/s. ~2176 total partitions merged to 12.
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.usagecounters2 0a2fa7d0-7afb-11ef-b677-bb2e6950de47] Compacted 21 sstables to [/var/lib/scylla/data/pc/usagecounters2-947b0b609f2c11e9bc5c000000000000/me-3gju_0dw9_4syuo2rw6a1kw0kvuv-big-Data.db:level=0]. 1MB to 1MB (~88% of original) in 243ms = 7MB/s. ~3200 total partitions merged to 638.
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.businessobjectentries4 0a555840-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4xgw02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4syup2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4tm002rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4jys02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/businessobjectentries4-0e6da7e0757d11e98f28000000000000/me-3gju_0dw9_4elvk2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.outgoingintegrationstates 0a555841-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_1rno02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_4pje82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_4rw9s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_5cwg02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_5999s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw8_2zlgg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw8_2vya82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw8_4vbq82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw8_4l1cw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw9_1pii82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dw8_0kd0w2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.tntindex2 0a555842-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_4vjg02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_51r7k2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_574402rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_3io6o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_3pbds2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_4rok02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_3klmo2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_4ztrk2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dwa_05km82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_53gxs2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_4i1c02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw9_3ktcg2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction,/var/lib/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gju_0dw4_4yjgx2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.outgoingintegrationstates 0a555841-7afb-11ef-b677-bb2e6950de47] Compacted 11 sstables to [/var/lib/scylla/data/pc/outgoingintegrationstates-6d4884207c8e11e98f26000000000000/me-3gju_0dwa_0bcy92rw6a1kw0kvuv-big-Data.db:level=0]. 175kB to 100kB (~57% of original) in 92ms = 1MB/s. ~1408 total partitions merged to 46.
Sep 25 05:00:10 o-p-L10-1 scylla[413744]: [shard 4:comp] compaction - [Compact pc.lock4 0a666f40-7afb-11ef-b677-bb2e6950de47] Compacting [/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw8_5r1pc2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_5grc02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_0bcy82rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_5etw02rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_2x0v42rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dwa_0902o2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_3gj0w2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_4ee5s2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_3elkw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_012kw2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_2my7k2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_4btkg2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_1cfts2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw9_28dio2rw6a1kw0kvuv-big-Data.db:level=0:origin=memtable,/var/lib/scylla/data/pc/lock4-e3f1e300676c11e9b751000000000000/me-3gju_0dw8_5djlc2rw6a1kw0kvuv-big-Data.db:level=0:origin=compaction]

```

The scylla-version running is 6.0.2-0.20240703.c9cd171f426e-1.

For some reasons the memtable-I/O started to freak out on a few nodes and the memtables-shares went up. Our application behavior has not changed and we have the same load as usual.

Restarting the whole cluster (every node, one by one) seems to have temporarily solved the issue, but we’re simply unsure about the root-cause and how to get there.

Maybe someone also seen this pattern appearing or has an idea what we’re seeing here?

Adding some screenshots of the Scylla-Advanced Dashboard.

 ![memtable_20240925](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/b/b29261f648f25293c5dcbc84aaa5ea4ed08a825c.png)

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 25, 2024, 4:15pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/2 "2024-09-25T16:15:20Z")

</div>

Might this issue be caused by #16514 ?

Stability: Off-strategy compaction is used to make sstables conform to the compaction strategy after an operation such as repair. Off-strategy compaction for TWCS will now have less storage space overhead. [#16514]

---

<div class="post-metadata">

**Author:** ![Womsel](https://avatars.discourse-cdn.com/v4/letter/w/47e85d/32.png) [@Womsel](https://forum.scylladb.com/u/Womsel)\
**Post date:** [September 26, 2024, 7:05pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/3 "2024-09-26T19:05:03Z")

</div>

Could this trigger anything, that might result in this weird memtable-compaction above?

From the logs:

Sep 26 16:22:30 o-p-L3-3 scylla[1381236]: [shard 5:stmt] lsa - Standard allocator failure, increasing head-room in section 0x604006144670 to 2048 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f  
--------  
seastar::internal::coroutine\_traits\_base::promise\_type  
Sep 26 16:22:30 o-p-L3-3 scylla[1381236]: [shard 5:stmt] lsa - Standard allocator failure, increasing head-room in section 0x604006144670 to 4096 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f  
--------  
seastar::internal::coroutine\_traits\_base::promise\_type  
Sep 26 16:22:30 o-p-L3-3 scylla[1381236]: [shard 5:stmt] lsa - Standard allocator failure, increasing head-room in section 0x604006144670 to 8192 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f  
--------  
seastar::internal::coroutine\_traits\_base::promise\_type  
Sep 26 16:22:30 o-p-L3-3 scylla[1381236]: [shard 5:stmt] lsa - Standard allocator failure, increasing head-room in section 0x604006144670 to 16384 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f

---

<div class="post-metadata">

**Author:** ![Womsel](https://avatars.discourse-cdn.com/v4/letter/w/47e85d/32.png) [@Womsel](https://forum.scylladb.com/u/Womsel)\
**Post date:** [September 27, 2024, 11:24am UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/4 "2024-09-27T11:24:24Z")

</div>

Interestingly, during the “Compaction Storm” we see alot of very small memtable flushes and compactions.

 ![2024-09-27_13-22_pendingflushes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/5/5bb71f81298f882181d7422400bb7648d574be24.png)

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 27, 2024, 1:21pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/5 "2024-09-27T13:21:04Z")

</div>

An even better screenshot “scylla\_memtables\_pending\_flushes\_bytes \> 0”:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/7/73b84c46b5e544f02c360313c6da44cb64248f74.png)

During the times of massive compactions occurring, it showed many small memtable flushes.

Question is: What can suddenly cause many small flushes to occur?

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [September 29, 2024, 1:15pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/6 "2024-09-29T13:15:53Z")

</div>

I think we had a problem with repair flushing at the beginning. Does time where the problem start match the start or a repair?

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 29, 2024, 3:13pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/7 "2024-09-29T15:13:20Z")

</div>

I was thinking about repairs triggering the flushes too, but it does not look like it (see graphs below).

But we have found something weird, that I think must be either a bug in the metrics, or perhaps some kinf of memory leak:

> <https://github.com/scylladb/scylladb/issues/20862>
>
> This is Scylla's bug tracker, to be used for reporting bugs only.
> If you have a… question about Scylla, and not a bug, please ask it in
> our mailing-list at scylladb-dev@googlegroups.com or in our slack channel.
> 
> \- \[x\] I have read the disclaimer above, and I am reporting a suspected malfunction in Scylla.
> 
> \*Installation details\*
> Scylla version (or git commit hash): 6.0.2
> Cluster size: Same issue with 6 node or 20 node cluster
> OS (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu 22.04
> 
> 
> The metric "scylla\_commitlog\_memory\_buffer\_bytes" seems to be ever growing since we have updated to 6.0.2:
> !\[image\](https://github.com/user-attachments/assets/b4121b84-6b36-40c9-9453-261a3b07a365)
> 
> 9/22 is the day the cluster was upgraded from 5.4 to 6.0. The times when the lines drop to 0 is when the cluster got restarted. This is the case on various different landscapes.
> 
> The metric is defined as
> \`sm::make\_gauge("memory\_buffer\_bytes", totals.buffer\_list\_bytes, sm::description("Holds the total number of bytes in internal memory buffers.")),\`

Graphs for the 24th:

Repairs:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/0/027defb6c29fbe5e61e4578d85a76f0085b12a26.png)

Memtable ioops:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/b/b394c265d553f56a7191768acddf23e84fca65cb.jpeg)

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 29, 2024, 3:15pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/8 "2024-09-29T15:15:13Z")

</div>

Regarding the github issue:

It seems the scylla\_commitlog\_memory\_buffer\_bytes is constantly going up since we updated to 6.0. Perhaps if that fillls up , it keeps flushing?

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [September 29, 2024, 3:28pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/9 "2024-09-29T15:28:54Z")

</div>

It’s probably related.

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 30, 2024, 12:18pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/10 "2024-09-30T12:18:34Z")

</div>

elcallio says that the commitlog-buffer-bytes cannot be responsible for any flushes.

I wonder what else can cause lots and lots of tiny flushes:

 ![flushes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/5/598708b18cae0b4eb000ccfb52ca539082e72345.png)

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [September 30, 2024, 2:27pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/11 "2024-09-30T14:27:47Z")

</div>

Please share commitlog metrics.

Also: does `nodetool flush` help?

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [September 30, 2024, 7:35pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/12 "2024-09-30T19:35:49Z")

</div>

Also we are quite a lot of memory related warnings:

```auto
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 262144 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 524288 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 1048576 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 2097152 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 4194304 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 8388608 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 16777216 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 33554432 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 67108864 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type
Sep 30 18:02:09 o-p-L3-6 scylla[3457277]: [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603003fc4670 to 134217728 [B]; trace: 0x645df8e 0x645e5a0 0x645e888 0x215033f 0x20f504e 0x20b6c0c 0x20b7000 0x1caeb23 0x1cadc83 0x1cab21d 0x1c70767 0x1c6b81d 0x1b8c108 0x47dde0a 0x144ccda 0x5f56c3f 0x5f57f27 0x5f7be90 0x5f1734a /opt/scylladb/libreloc/libc.so.6+0x8c946 /opt/scylladb/libreloc/libc.so.6+0x11296f
                                                       --------
                                                       seastar::internal::coroutine_traits_base<void>::promise_type

```

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [October 1, 2024, 9:35am UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/13 "2024-10-01T09:35:53Z")

</div>

My previous comment is still waiting for approval.

One metrics I find particular interesting is:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/2/2068b6ed0143934752c56b519165706180e0d1e6.png)

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [October 2, 2024, 7:54pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/14 "2024-10-02T19:54:09Z")

</div>

Another idea: Can the schema-commitlog cause flushes of our data tables?

For some reason the scylla\_schema\_commitlog\_active\_allocations was pretty high after the 6.0 upgrade. On 25th when we did a rolling restart it went down a lot and stayed like that ever since:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/f/f68d37fcac96fe087cef04dbdc26e53b39384259.png)

Is it possible that some action was still pending from the upgrade, that caused many schema commitlog activity?

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [October 2, 2024, 8:32pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/15 "2024-10-02T20:32:56Z")

</div>

I do not think that nodetool flush helps, as it seems it was flushing like crazy anyway. Also I am 99% sure we did some flushes in between and it did not help. Only the rolling restart did. I have no cluster where I can reproduce it, so I cannot say 100% for sure.

Lots of small flushes:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/9/90e41331224b064eefb71ac40d0f2e1fabfe0066.png)

As for commitlog metrics. Here are the metrics around the time when the compactions started going up (~ 24th 11:00 UTC):

“Merged from Memtable to Cache” is going up for the hosts having the issue:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/2/2068b6ed0143934752c56b519165706180e0d1e6.png)  
I cannot say if that is cause or symptom.

Memtable switches goes up for the hot tables:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/c/c1128ce7a6b2a32e8e5639c3b98291d7d647b8d7.png)

Schema commitlog latency goes up on some of the hosts, which I think is a symptom (IO being bottlenecked):

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/7/745d9378b0d4b82889c119a39e5088c0853f7e7d.png)

What I find interesing is that the “reserved disk space” is similar in size

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/5/50ee2d522bdf097d1c07c3a72398caa3a66c1213.png)

Commitlog buffer size (this is what should be fixed in the ticket) is similar in size. My initial theory was, that this puts pressure on scylla to flush to free up memory. But elcallio does not think so:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/9/99ca0fb25957a52a03c5a2aac02ad6f98827e3f3.png)

Let me know if you are looking for something in particular…

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [October 6, 2024, 10:59am UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/16 "2024-10-06T10:59:17Z")

</div>

You can try running with the commitlog logger at debug mode. I’m looking at this log line:

```c++
    clogger.debug("Flushing ({} MB) to {}", flushing/(1024*1024), high);

```

If we correlate it with the memtable flush activity, we can focus on commitlog, and if not, it’s somewhere else.

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [October 6, 2024, 11:02am UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/17 "2024-10-06T11:02:14Z")

</div>

Maybe that’s the answer. We keep increasing the memory reserve. But it’s not quite right, since the memtable size isn’t affected by the reserves.

In any case, nothing can work like this. Suggest looking at the system.large\* tables to look for large rows/cells/collections (large partitions are fine).

Also, please decode the backtraces.

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [October 6, 2024, 5:12pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/18 "2024-10-06T17:12:11Z")

</div>

I would say its definetaly related to flushes:

 ![image](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/a/aaaebf90aad3b580b2d982c8e5bf53d4e103764f.png)

I guess the question is: Were these flushes triggered by commitlog or memory?

---

<div class="post-metadata">

**Author:** ![horschi](https://avatars.discourse-cdn.com/v4/letter/h/a6a055/32.png) [@horschi](https://forum.scylladb.com/u/horschi)\
**Post date:** [October 6, 2024, 5:38pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/19 "2024-10-06T17:38:02Z")

</div>

As for large\* …

- We basically do not use **collections**. So I would rule that out. I must admit I do not know how to check for them, as there is no large\_collections table.
- There are quite some large **cells** up to 3MB. But it cannot go larger than that as our application checks for that and throws exception during write.
- No large **rows** reported.
- **Partitions** : some go up to 1GB and 25m rows. I would hope that this is not a problem.

---

<div class="post-metadata">

**Author:** ![avikivity](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/avikivity/32/38_2.png) [@avikivity](https://forum.scylladb.com/u/avikivity)\
**Post date:** [October 6, 2024, 7:15pm UTC](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958/20 "2024-10-06T19:15:22Z")

</div>

Backtrace decode may help here

[Next page](https://forum.scylladb.com/t/compaction-storm-slows-down-scylla/2958.md?page=2)