Hello Team,
We are facing issue of old nodes getting crashed after adding new nodes.
Details :
Scylla version : 6.1.2-0.20240915.b60f9ef4c223
Backup restoration performed on 4 node scylla cluster running on 6.1.
Source :
Scylla version : 5.2.19
Scylla manager : 3.2
Nodes : 4
Size : 1.5 TB each node
Destination :
Scylla version : 6.1.2
Scylla manager : 3.3
Nodes : 4
Tablets enabled cluster.
Restoration and repair post restoration was successfully completed.
Once 4 node cluster of 6.1 was running with Size 1.5 TB each node, we tried doing elastic scaling by adding 4 more nodes by starting scylla-server service simultaneously.
All 4 new nodes got added in 2 mins by coming UN state but with minimal 4-5 GB of data.
Tablets redistribution started.
While redistribution was going on, observed that we old two nodes got abruptly crashed and not able to come back up.
Attaching logs here of one of the nodes which got crashed.
Oct 11 12:37:27 NODENAME scylla[60269]: [shard 0:main] table - Unable to load SSTable /var/lib/scylla/data/vss/embeddings-51d0492086fc11ef8be67886065e5abf/me-3gk9_1nef_1r88g2c1uc12qoe97m-big-Data.db that belongs to tablets 2 and 3, at: 0x5e9e>
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::coroutine::parallel_for_each<replica::distributed_loader::populate_keyspace(seastar::sharded<replica::database>&, seastar::sharded<db::system_keyspace>&, replica::keyspace&, seastar::basic_sst>
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::coroutine::parallel_for_each<replica::distributed_loader::populate_keyspace(seastar::sharded<replica::database>&, seastar::sharded<db::system_keyspace>&, replica::keyspace&, seastar::basic_sst>
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::internal::complete_when_all<seastar::internal::extract_values_from_futures_vector<seastar::future<void> >, seastar::futur>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::discard_result()::{lambda((auto:1&&)...)#1}, seastar::future<void>::then_impl_nrvo<seastar::future<void>::d>
--------
seastar::(anonymous namespace)::thread_wake_task
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::async<replica::distributed_loader::init_non_system_keyspaces(seastar::sharded<replica::database>&, seastar::sharded<servi>
--------
seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<seastar::async<replica::distributed_loader::init_non_system_keyspaces(seastar::sharded<replica>
--------
seastar::(anonymous namespace)::thread_wake_task
--------
seastar::continuation<seastar::internal::promise_base_with_type<int>, seastar::async<scylla_main(int, char**)::$_0::operator()() const::{lambda()#2}>(seastar::thread_attributes, scylla_main(int, char**>
--------
seastar::continuation<seastar::internal::promise_base_with_type<int>, seastar::future<int>::finally_body<seastar::async<scylla_main(int, char**)::$_0::operator()() const::{lambda()#2}>(seastar::thread_>
--------
seastar::continuation<seastar::internal::promise_base_with_type<int>, seastar::future<int>::finally_body<seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&)::$_0::operator>
Oct 11 12:37:27 NODENAME scylla[60269]: terminate called after throwing an instance of 'seastar::internal::backtraced<std::runtime_error>'
Oct 11 12:37:27 NODENAME scylla[60269]: what(): Unable to load SSTable /var/lib/scylla/data/vss/embeddings-51d0492086fc11ef8be67886065e5abf/me-3gk9_1nef_1r88g2c1uc12qoe97m-big-Data.db that belongs to tablets 2 and 3 Backtrace: 0x5e9e22e 0x5e>
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::internal::coroutine_traits_base<void>::promise_type
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::internal::coroutine_traits_base<void>::promise_type
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::internal::coroutine_traits_base<void>::promise_type
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::coroutine::parallel_for_each<replica::distributed_loader::populate_keyspace(seastar::sharded<replica::database>&, seastar::sharded<db::system_keyspace>&, replica::keyspace&, seastar::basic_sst>
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::internal::coroutine_traits_base<void>::promise_type
Oct 11 12:37:27 NODENAME scylla[60269]: --------
Oct 11 12:37:27 NODENAME scylla[60269]: seastar::coroutine::parallel_for_each<replica::distributed_loader::populate_keyspace(seastar::sharded<replica::database>&, seastar::sharded<db::system_keyspace>&, replica::keyspace&, seastar::basic_sst>
Oct 11 12:33:35 NODENAME scylla[18799]: 0x13842a5
Oct 11 12:33:35 NODENAME scylla[18799]: 0x1385c60
Oct 11 11:40:01 NODENAME scylla[18799]: [shard 0:comp] compaction - [Compact system.peers 8d28e7b0-87c5-11ef-9987-878599d64622] Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/me-3gka_0wep_4yywh2c1u>
Oct 11 11:40:01 NODENAME scylla[18799]: [shard 0:comp] compaction - [Compact system.group0_history 8d2a9560-87c5-11ef-9987-878599d64622] Compacting [/var/lib/scylla/data/system/group0_history-027e42f5683a3ed7b404a0100762063c/me-3gka_055x_02sb>
Oct 11 11:40:01 NODENAME scylla[18799]: [shard 0:comp] sstable - Rebuilding bloom filter /var/lib/scylla/data/system/group0_history-027e42f5683a3ed7b404a0100762063c/me-3gka_0wep_51jhs2c1uc12qoe97m-big-Filter.db: resizing bitset from 328 bytes>
Oct 11 11:40:01 NODENAME scylla[18799]: [shard 0:comp] compaction - [Compact system.group0_history 8d2a9560-87c5-11ef-9987-878599d64622] Compacted 2 sstables to [/var/lib/scylla/data/system/group0_history-027e42f5683a3ed7b404a0100762063c/me-3>
Oct 11 11:40:02 NODENAME scylla[18799]: [shard 0:strm] stream_session - [Stream #779698c4-87c5-11ef-bc3c-ec5f9328e09e] Streaming plan for Tablet migration-vss-index-0 succeeded, peers={10.138.64.129}, tx=1205197 KiB, 33087.57 KiB/s, rx=0 KiB,>
Oct 11 11:40:02 NODENAME scylla[18799]: [shard 30:strm] table - Cleaned up tablet 0 of table vss.catalog_realtime_accumulator successfully.
Oct 11 11:40:02 NODENAME scylla[18799]: [shard 12:strm] table - Cleaned up tablet 23 of table vss.catalog_realtime_accumulator successfully.
Oct 11 12:31:43 NODENAME scylla[18799]: [shard 0:comp] compaction - [Compact system.peers c59f6310-87cc-11ef-9987-878599d64622] Compacted 2 sstables to [/var/lib/scylla/data/system/peers-37f71aca7dc2383ba70672528af04d4f/me-3gka_0ysv_08scx2c1u>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 0: gms] table - Detected tablet split for table vss.embeddings, increasing from 64 to 128 tablets
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 0: gms] table - Found that storage of group 1 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e >
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 1: gms] table - Found that storage of group 34 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 3: gms] table - Found that storage of group 50 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 29: gms] table - Found that storage of group 13 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 20: gms] table - Found that storage of group 46 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 6: gms] table - Found that storage of group 4 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e >
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 26: gms] table - Found that storage of group 24 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 7: gms] table - Found that storage of group 59 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 27: gms] table - Found that storage of group 19 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 24: gms] table - Found that storage of group 31 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 10: gms] table - Found that storage of group 38 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 11: gms] table - Found that storage of group 29 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 25: gms] table - Found that storage of group 27 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 22: gms] table - Found that storage of group 41 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 23: gms] table - Found that storage of group 32 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 13: gms] table - Found that storage of group 15 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 12: gms] table - Found that storage of group 23 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: Aborting on shard 0, in scheduling group gossip.
Oct 11 12:33:35 NODENAME scylla[18799]: Backtrace:
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59d7144
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59965bb
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59cbf16
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x40cff
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x994a3
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x40c4d
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x28901
Oct 11 12:33:35 NODENAME scylla[18799]: 0x3e22ba7
Oct 11 12:33:35 NODENAME scylla[18799]: 0x13d93ea
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59a691f
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59a7e8a
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59a9077
Oct 11 12:33:35 NODENAME scylla[18799]: 0x59a8428
Oct 11 12:33:35 NODENAME scylla[18799]: 0x5938773
Oct 11 12:33:35 NODENAME scylla[18799]: 0x5937ad3
Oct 11 12:33:35 NODENAME scylla[18799]: 0x13842a5
Oct 11 12:33:35 NODENAME scylla[18799]: 0x1385c60
Oct 11 12:33:35 NODENAME scylla[18799]: 0x13826c3
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x2a087
Oct 11 12:33:35 NODENAME scylla[18799]: /opt/scylladb/libreloc/libc.so.6+0x2a14a
Oct 11 12:33:35 NODENAME scylla[18799]: 0x137fd44
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 5: gms] table - Found that storage of group 21 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 15: gms] table - Found that storage of group 62 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 14: gms] table - Found that storage of group 6 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e >
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 28: gms] table - Found that storage of group 17 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 9: gms] table - Found that storage of group 45 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 30: gms] table - Found that storage of group 8 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e >
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 31: gms] table - Found that storage of group 11 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 18: gms] table - Found that storage of group 55 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 4: gms] table - Found that storage of group 37 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 21: gms] table - Found that storage of group 43 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 8: gms] table - Found that storage of group 52 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 17: gms] table - Found that storage of group 56 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 16: gms] table - Found that storage of group 61 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn't split correctly, therefore groups cannot be remapped with the new tablet count., at: 0x5e9e22e>
--------
seastar::smp_message_queue::async_work_item<seastar::sharded<service::storage_service>::invoke_on_all(seastar::smp_submit_to_options, std::function<seastar::future<void> (service::storage_service&)>)::>
Oct 11 12:33:35 NODENAME scylla[18799]: [shard 0: gms] storage_service - Failed to apply token_metadata changes: seastar::internal::backtraced<std::runtime_error> (Found that storage of group 1 for table 51d04920-86fc-11ef-8be6-7886065e5abf w>
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type
--------
seastar::internal::coroutine_traits_base<void>::promise_type). Aborting.
Oct 11 12:37:21 NODENAME systemd[1]: scylla-server.service: Main process exited, code=dumped, status=6/ABRT
Oct 11 12:37:21 NODENAME systemd[1]: scylla-server.service: Failed with result 'core-dump'.
Oct 11 12:37:22 NODENAME systemd[1]: scylla-server.service: Scheduled restart job, restart counter is at 1.
Oct 11 12:37:22 NODENAME systemd[1]: Stopped Scylla Server.
Oct 11 12:37:22 NODENAME systemd[1]: Starting Scylla Server...
Oct 11 12:37:22 NODENAME scylla[60269]: Scylla version 6.1.2-0.20240915.b60f9ef4c223 with build-id c713ac9e819492d7560aa3ad461c43cf404c977b starting ...
Oct 11 12:37:22 NODENAME scylla[60269]: command used: "/usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties-file=/etc/scylla.d/io_properties.yaml --lock-memory=1"
Oct 11 12:37:22 NODENAME scylla[60269]: pid: 60269
Oct 11 12:37:22 NODENAME scylla[60269]: parsed command line options: [log-to-syslog, (positional) 1, log-to-stdout, (positional) 0, default-log-level, (positional) info, network-stack, (positional) posix, io-properties-file: /etc/scylla.d/io_pr>
Oct 11 12:37:22 NODENAME scylla[60269]: seastar - Reactor backend: linux-aio
Oct 11 12:37:23 NODENAME scylla[60269]: seastar - Perf-based stall detector creation failed (EACCESS), try setting /proc/sys/kernel/perf_event_paranoid to 1 or less to enable kernel backtraces: falling back to posix timer.
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 3:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 10:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 6:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 20:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 29:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 11:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 2:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 27:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 9:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 5:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 4:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 1:main] seastar - updated: blocked-reactor-notify-ms=36000000000
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - Option is deprecated : force_schema_commit_log
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - installing SIGHUP handler
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - Scylla version 6.1.2-0.20240915.b60f9ef4c223 with build-id c713ac9e819492d7560aa3ad461c43cf404c977b starting ...
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting API server
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting prometheus API server
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - creating snitch
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] snitch_logger - GCESnitch using region: asia-southeast1, zone: b.
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting tokens manager
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting effective_replication_map factory
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting migration manager notifier
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting per-shard database core
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - creating and verifying directories
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] init - starting compaction_manager
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 0:main] task_manager - Registered module compaction
Oct 11 12:37:24 NODENAME scylla[60269]: [shard 7:main] task_manager - Registered module compaction
Tried taking service reboot but didnt helped.
Any resolution for such cases where [shard 5: gms] table - Found that storage of group 21 for table 51d04920-86fc-11ef-8be6-7886065e5abf wasn’t split correctly, therefore groups cannot be remapped with the new tablet count.
Let us know if anything is needed for debugging ?