Nodes become unusable when SPLIT compactions are going on

ScyllaDB
,
1

Hello,

I recently started migrating my production cluster from Cassandra 4 to Scylla 6.0.2. After having had a great success with our dev cluster, we decided to go to prod.
I switched the stack around 10 days ago, and started copying data over.
I’m currently facing two problems:

  1. Underwhelming performance for the data copy (custom ETL that writes unlogged batches)
  2. Every once in a while (around 2 or 3 days) the table that I’m currently copying will start a bunch of SPLIT compactions (probably tablets being split?) and when that happens, the cluster becomes basically unusable until all of them finish. Latencies spike thru the roof and I start getting timeouts left and write.

My keyspace has tablets enabled.

My production workload is currently writing around 60k writes/sec at peak hours, and around 10k at off-peak hours.
My copy program is able to copy at around 200k rows/s for the current table (speed varies from table to table).
I have around 6TB of data to be copied.
Machines are aws m6a.2xlarge (8c, 32GB ram) with 4 TB of EBS storage, 12k provisioned IOPS, 1000 MiB/s provisioned throughput. Cluster currently have 8 nodes.

I have tried a bunch of things to fix this problem, but none of them seem to fix it.
I’ve tried:

  • increasing io-latency-goal-ms to 2ms and then 10ms
  • manually changing io_properties.yaml to 12k IOPS and 1000MiB/s bandwidth
  • increasing max-task-backlog to 5000
  • setting compaction throughput limit to 40MB/s

During data copy, I will observe that the metrics “Writes currently blocked on commitlog” and “Writes blocked on commitlog” will go high, and I’ll have to reduce the copy throuput until I don’t see anything on these graphs anymore.
When the split compactions fire and the cluster gets unusable, I’ll only see “Writes timed out” and “Writes failed”.

EBS graphs don’t show usage above 600 IOPS at any time, despite being provisioned for 12000. Bandwidth also won’t go above 40MB/s, despite being provisioned for 1000.

CPU does not appear to be maxed.

What can I do to diagnose the problem with the compactions? the underwhelming throughput on the copy is bad, but tolerable since it’s a one off thing, but these SPLIT compactions keep bringing my production pipeline down for many hours at a time and I don’t know what to do.

2

One node just restarted while doing a SPLIT compaction. this was in the logs:

INFO  2024-08-14 21:09:36,419 [shard 1:comp] compaction - [Compact sessions.raw_sensors2 00615990-5a78-11ef-bb5f-ce6e2d321687] Compacting of 2 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at 0x645e7ce 0x645ede0 0x645f0c8 0x26585ac 0x6335c76
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<void>::then_impl_nrvo<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<sstables::compaction_result> >(sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#2}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>, seastar::future<sstables::compaction_result>::then_wrapped_nrvo<seastar::future<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false> >(seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#3}, false>&, seastar::future_state<sstables::compaction_result>&&)#1}, sstables::compaction_result>
   --------
   seastar::internal::coroutine_traits_base<sstables::compaction_result>::promise_type
   --------
   seastar::internal::coroutine_traits_base<std::optional<sstables::compaction_stats> >::promise_type
   --------
   seastar::shared_future<std::optional<sstables::compaction_stats> >::shared_state
ERROR 2024-08-14 21:09:36,419 [shard 1:comp] compaction_manager - Compaction task 0x60100addd500 for table sessions.raw_sensors2 compaction_group=20 [0x60100602b120]: failed: std::bad_alloc (std::bad_alloc). Will retry in 5 seconds
INFO  2024-08-14 21:09:39,227 [shard 1:strm] compaction - [Split sessions.raw_sensors2 fc1ecde0-5a77-11ef-bb5f-ce6e2d321687] Splitting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at 0x645e7ce 0x645ede0 0x645f0c8 0x26585ac 0x6335c76
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<void>::then_impl_nrvo<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<sstables::compaction_result> >(sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#2}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>, seastar::future<sstables::compaction_result>::then_wrapped_nrvo<seastar::future<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false> >(seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#3}, false>&, seastar::future_state<sstables::compaction_result>&&)#1}, sstables::compaction_result>
   --------
   seastar::internal::coroutine_traits_base<sstables::compaction_result>::promise_type
   --------
   seastar::internal::coroutine_traits_base<sstables::compaction_result>::promise_type
   --------
   seastar::internal::coroutine_traits_base<sstables::compaction_result>::promise_type
   --------
   seastar::internal::coroutine_traits_base<std::optional<sstables::compaction_stats> >::promise_type
   --------
   seastar::shared_future<std::optional<sstables::compaction_stats> >::shared_state
ERROR 2024-08-14 21:09:39,227 [shard 1:strm] compaction_manager - Split task 0x60100a970000 for table sessions.raw_sensors2 compaction_group=21 [0x60100602b0c0]: failed: std::bad_alloc (std::bad_alloc). Will retry in 5 seconds
INFO  2024-08-14 21:09:41,420 [shard 1:comp] compaction - [Compact sessions.raw_sensors2 85d22ec0-5a81-11ef-bb5f-ce6e2d321687] Compacting [/var/lib/scylla/data/sessions/raw_sensors2-a71cee60503311ef9bc266588bcb699a/me-3gio_1eeh_2inw02uoq8w7yvw8qp-big-Data.db:level=0:origin=split,/var/lib/scylla/data/sessions/raw_sensors2-a71cee60503311ef9bc266588bcb699a/me-3gio_1dr1_46go02uoq8w7yvw8qp-big-Data.db:level=0:origin=split]
INFO  2024-08-14 21:09:42,289 [shard 1:comp] compaction - [Compact sessions.raw_sensors2 85d22ec0-5a81-11ef-bb5f-ce6e2d321687] Compacting of 2 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at 0x645e7ce 0x645ede0 0x645f0c8 0x26585ac 0x6335c76
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<void>::then_impl_nrvo<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#2}, seastar::future<sstables::compaction_result> >(sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#2}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
   --------
   seastar::continuation<seastar::internal::promise_base_with_type<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>, seastar::future<sstables::compaction_result>::then_wrapped_nrvo<seastar::future<sstables::compaction_result>, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false> >(seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#3}, false>&&)::{lambda(seastar::internal::promise_base_with_type<sstables::compaction_result>&&, seastar::future<sstables::compaction_result>::finally_body<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, auto:1&&, (auto:2&&)...)::{lambda()#3}, false>&, seastar::future_state<sstables::compaction_result>&&)#1}, sstables::compaction_result>
   --------
   seastar::internal::coroutine_traits_base<sstables::compaction_result>::promise_type
   --------
   seastar::internal::coroutine_traits_base<std::optional<sstables::compaction_stats> >::promise_type
   --------
   seastar::shared_future<std::optional<sstables::compaction_stats> >::shared_state
ERROR 2024-08-14 21:09:42,289 [shard 1:comp] compaction_manager - Compaction task 0x60100addd500 for table sessions.raw_sensors2 compaction_group=20 [0x60100602b120]: failed: std::bad_alloc (std::bad_alloc). Will retry in 10 seconds
ERROR 2024-08-14 21:09:43,283 [shard 1:strm] rpc - client 100.111.65.76:53297: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,285 [shard 1:strm] rpc - client 100.103.121.206:61385: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,290 [shard 1:strm] rpc - client 100.111.65.76:63425: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,297 [shard 1:strm] rpc - client 100.111.65.76:61337: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,331 [shard 1:strm] rpc - client 100.103.121.206:53769: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,333 [shard 1:strm] rpc - client 100.96.228.16:55273: server connection dropped: std::bad_alloc
ERROR 2024-08-14 21:09:43,335 [shard 1:strm] rpc - client 100.111.65.76:58577: server connection dropped: std::bad_alloc
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborting on shard 1.
Backtrace:
  0x5f45848
  0x5f7c0d1
  /opt/scylladb/libreloc/libc.so.6+0x3dbaf
  /opt/scylladb/libreloc/libc.so.6+0x8e883
  /opt/scylladb/libreloc/libc.so.6+0x3dafd
  /opt/scylladb/libreloc/libc.so.6+0x2687e
  /opt/scylladb/libreloc/libstdc++.so.6+0xa4d38
  /opt/scylladb/libreloc/libstdc++.so.6+0xb4f6b
  /opt/scylladb/libreloc/libstdc++.so.6+0xb4fd6
  0x13d88fa
  0x38d4fb3
  0x38e48fa
  0x38e41fd
  0x1adf93b
  0x1b76ff1
  0x1b9772b
  0x1b972f8
  0x5eed862
  0x5f5747f
  0x5f58767
  0x5f7c6d0
  0x5f17b8a
  /opt/scylladb/libreloc/libc.so.6+0x8c946
  /opt/scylladb/libreloc/libc.so.6+0x11296f
2024-08-14 21:09:45,479 INFO exited: scylla (terminated by SIGABRT (core dumped); not expected)
3

@matheus2740 do you have monitoring data at the time of the incident? logs can also help. You can upload the requested data using these instructions: How to Report a ScyllaDB Problem | ScyllaDB Docs

1 Like
4

Split happens in the same I/O class as maintenance ops like repair / streaming. So you can try tuning stream_io_throughput_mb_per_sec.

5

@raphaelsc I got the same recommendation about throttling streaming in slack some weeks ago, and it seems to have mitigated the issue. it isn’t a fix, but at least the cluster stopped crashing.

Monitoring data can be found at the github ticket: Cluster becomes unusable when SPLIT compactions are happening · Issue #20211 · scylladb/scylladb · GitHub

Unfortunately I don’t have more than that since our log/metric retention is not that long and it’s been a while.