When I perform removenode operation on a node with permanent DN, other nodes DN are gone. I see the following abnormal information on the DN node:
storage_proxy - Exception when communicating with 10.6.0.2, to read from system.paxos: logalloc::bad_alloc (failed to refill emergency reserve of 30 (have 20 free segments))
Does this result in a node DN?
The to read from system.paxos makes me think this is related to LWT and since we don’t use LWT internally, therefore this error shouldn’t cause a node to be DN.
This seems like a simple user read failure.
I saw that Scylla threw the following message before DN
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
Aborting on shard 19.
Backtrace:
0x5bc3f38
0x5bfa082
/opt/scylladb/libreloc/libc.so.6+0x3dbaf
/opt/scylladb/libreloc/libc.so.6+0x8e883
/opt/scylladb/libreloc/libc.so.6+0x3dafd
/opt/scylladb/libreloc/libc.so.6+0x2687e
/opt/scylladb/libreloc/libstdc++.so.6+0xa4d18
/opt/scylladb/libreloc/libstdc++.so.6+0xb4f4b
/opt/scylladb/libreloc/libstdc++.so.6+0xb4fb6
0x132ec3a
0x60ab702
0x18a8423
0x18a583c
0x44a5459
0x43f845d
0x440adf4
0x4408365
0x5fb3706
2024-10-21 09:34:50,720 INFO exited: scylla (terminated by SIGABRT; not expected)
2024-10-21 09:34:51,724 INFO spawned: 'scylla' with pid 537463
The decode information is as follows:
/usr/lib/debug/opt/scylladb/libexec/scylla-5.4.10-2.20241022.c9dcdbfcfe4c.x86_64.debug: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 3.2.0, BuildID[sha1]=fd75a553ef426894076f70b0ee274a0c6584a1b3, not stripped
Backtrace:
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:64
(inlined by)seastar::backtrace_buffer::append_backtrace() at ./build/release/seastar/./seastar/src/core/reactor.cc:825
(inlined by)seastar::print_with_backtrace(seastar::backtrace_buffer&, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:855
seastar::print_with_backtrace(char const*, bool) at ./build/release/seastar/./seastar/src/core/reactor.cc:867
(inlined by)seastar::sigabrt_action() at ./build/release/seastar/./seastar/src/core/reactor.cc:4031
(inlined by)operator() at ./build/release/seastar/./seastar/src/core/reactor.cc:4007
(inlined by)__invoke at ./build/release/seastar/./seastar/src/core/reactor.cc:4003
/opt/scylladb/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), BuildID[sha1]=7026fe8c129a523e07856d7c96306663ceab6e24, for GNU/Linux 3.2.0, stripped
addr2line: /opt/scylladb/libreloc/libc.so.6: don't know how to handle section `.relr.dyn' [0x 13]
__sigaction at ??:?
pthread_key_delete at ??:?
gsignal at ??:?
abort at ??:?
/opt/scylladb/libreloc/libstdc++.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=694f64a512ecd55e04f1587ddd790ec30cb0b726, stripped
__cxa_throw_bad_array_new_length at ??:?
std::rethrow_exception(std::__exception_ptr::exception_ptr) at ??:?
std::terminate() at ??:?
__clang_call_terminate at main.cc:?
void seastar::future<void>::schedule<seastar::internal::promise_base_with_type<void>, seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> seastar::future<void>::then_impl_nrvo<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&, seastar::future_state<seastar::internal::monostate>&&)#1}>(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&, seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&, seastar::future<void> seastar::future<void>::then_impl_nrvo<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&, seastar::future_state<seastar::internal::monostate>&&)#1}&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1268
(inlined by)seastar::future<void> seastar::future<void>::then_impl_nrvo<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1450
(inlined by)seastar::future<void> seastar::future<void>::then_impl<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1476
(inlined by)seastar::future<void> seastar::future<void>::then<seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0, seastar::future<void> >(seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&)::$_0&&) at ./build/release/seastar/./seastar/include/seastar/core/future.hh:1399
(inlined by)seastar::rpc::connection::stream_receive(seastar::circular_buffer<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > >, std::allocator<seastar::foreign_ptr<std::unique_ptr<seastar::rpc::rcv_buf, std::default_delete<seastar::rpc::rcv_buf> > > > >&) at ./build/release/seastar/./seastar/src/rpc/rpc.cc:563
operator() at /tmp/./seastar/include/seastar/rpc/rpc_impl.hh:845
seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}&>(seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)seastar::futurize<std::invoke_result<seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}>::type>::type seastar::smp::submit_to<seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}>(unsigned int, seastar::smp_submit_to_options, std::invoke_result&&) at /tmp/./seastar/include/seastar/core/smp.hh:385
(inlined by)seastar::futurize<std::invoke_result<seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}>::type>::type seastar::smp::submit_to<seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()()::{lambda()#2}>(unsigned int, std::invoke_result&&) at /tmp/./seastar/include/seastar/core/smp.hh:412
(inlined by)seastar::rpc::source_impl<netw::serializer, repair_row_on_wire_with_cmd>::operator()() at /tmp/./seastar/include/seastar/rpc/rpc_impl.hh:840
seastar::rpc::source<repair_row_on_wire_with_cmd>::operator()() at /tmp/./seastar/include/seastar/rpc/rpc_types.hh:361
(inlined by)operator() at /tmp/repair/row_level.cc:167
(inlined by)seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::invoke<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() const::{lambda()#1}&>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() const::{lambda()#1}&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)seastar::future<void> seastar::repeat<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() const::{lambda()#1}>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}::operator()() const::{lambda()#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:126
operator() at /tmp/repair/row_level.cc:165
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)auto seastar::futurize_invoke<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&) at /tmp/./seastar/include/seastar/core/future.hh:2037
(inlined by)seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>::operator()(seastar::future<void>&&) at /tmp/./seastar/include/seastar/core/future.hh:1631
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>, seastar::future<void> >(seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>&&, seastar::future<void>&&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_common<false, seastar::future<void>, seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true> >(seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>&&) at /tmp/./seastar/include/seastar/core/future.hh:1558
(inlined by)seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped_maybe_erase<false, seastar::future<void>, seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true> >(seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>&&) at /tmp/./seastar/include/seastar/core/future.hh:1515
(inlined by)seastar::futurize<seastar::future<void> >::type seastar::future<void>::then_wrapped<seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>, seastar::future<void> >(seastar::future<void>::finally_body<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}, true>&&) & at /tmp/./seastar/include/seastar/core/future.hh:1499
(inlined by)seastar::future<void> seastar::future<void>::finally<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}::operator()(unsigned int)::{lambda()#1}&&) at /tmp/./seastar/include/seastar/core/future.hh:1616
(inlined by)operator() at /tmp/repair/row_level.cc:162
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}, unsigned int>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&, unsigned int&&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)auto seastar::futurize_invoke<sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}, unsigned int>(sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&, unsigned int&&) at /tmp/./seastar/include/seastar/core/future.hh:2037
(inlined by)seastar::future<void> seastar::parallel_for_each<boost::range_detail::integer_iterator<unsigned int>, boost::range_detail::integer_iterator<unsigned int>, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}>(boost::range_detail::integer_iterator<unsigned int>, boost::range_detail::integer_iterator<unsigned int>, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:575
(inlined by)seastar::future<void> seastar::internal::parallel_for_each_impl<boost::integer_range<unsigned int>, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}>(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:628
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&), boost::integer_range<unsigned int>, {lambda(unsigned int)#1}>(seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&), boost::integer_range<unsigned int>&&, {lambda(unsigned int)#1}&&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)auto seastar::futurize_invoke<seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&), boost::integer_range<unsigned int>, {lambda(unsigned int)#1}>(seastar::future<void> (*&)(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&), boost::integer_range<unsigned int>&&, {lambda(unsigned int)#1}&&) at /tmp/./seastar/include/seastar/core/future.hh:2037
(inlined by)seastar::future<void> seastar::parallel_for_each<boost::integer_range<unsigned int>, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}>(boost::integer_range<unsigned int>&&, sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close()::{lambda(unsigned int)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:643
(inlined by)sink_source_for_repair<repair_hash_with_cmd, repair_row_on_wire_with_cmd>::close() at /tmp/repair/row_level.cc:159
(inlined by)repair_meta::stop() at /tmp/repair/row_level.cc:998
repair_meta::repair_row_level_stop(gms::inet_address, seastar::basic_sstring<char, unsigned int, 15u, true>, seastar::basic_sstring<char, unsigned int, 15u, true>, nonwrapping_interval<dht::token>) at /tmp/repair/row_level.cc:1758
(inlined by)operator() at /tmp/repair/row_level.cc:3737
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}, gms&>(row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&, gms&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)auto seastar::futurize_invoke<row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}, gms&>(row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&, gms&) at /tmp/./seastar/include/seastar/core/future.hh:2037
(inlined by)seastar::future<void> seastar::parallel_for_each<__gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, __gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}>(__gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, __gnu_cxx::__normal_iterator<gms::inet_address*, std::vector<gms::inet_address, std::allocator<gms::inet_address> > >, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:575
(inlined by)seastar::future<void> seastar::internal::parallel_for_each_impl<std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}>(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:628
(inlined by)seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, gms::inet_address const&>(seastar::future<void> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, gms::inet_address const&) at /tmp/./seastar/include/seastar/core/future.hh:2006
(inlined by)auto seastar::futurize_invoke<seastar::future<void> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, gms::inet_address const&>(seastar::future<void> (*&)(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&), std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, gms::inet_address const&) at /tmp/./seastar/include/seastar/core/future.hh:2037
(inlined by)seastar::future<void> seastar::parallel_for_each<std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}>(std::vector<gms::inet_address, std::allocator<gms::inet_address> >&, row_level_repair::run()::{lambda()#1}::operator()() const::{lambda(gms::inet_address const&)#1}&&) at /tmp/./seastar/include/seastar/core/loop.hh:643
(inlined by)operator() at /tmp/repair/row_level.cc:3735
void std::__invoke_impl<void, row_level_repair::run()::{lambda()#1}>(std::__invoke_other, row_level_repair::run()::{lambda()#1}&&) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61
(inlined by)std::__invoke_result<row_level_repair::run()::{lambda()#1}>::type std::__invoke<row_level_repair::run()::{lambda()#1}>(std::__invoke_result&&, (row_level_repair::run()::{lambda()#1}&&)...) at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96
(inlined by)_ZSt12__apply_implIZN16row_level_repair3runEvEUlvE_St5tupleIJEEJEEDcOT_OT0_St16integer_sequenceImJXspT1_EEE at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2288
(inlined by)_ZSt5applyIZN16row_level_repair3runEvEUlvE_St5tupleIJEEEDcOT_OT0_ at /usr/bin/../lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/tuple:2299
(inlined by)seastar::future<void> seastar::futurize<void>::apply<row_level_repair::run()::{lambda()#1}>(row_level_repair::run()::{lambda()#1}&&, std::tuple<>&&) at /tmp/./seastar/include/seastar/core/future.hh:1970
(inlined by)operator() at /tmp/./seastar/include/seastar/core/thread.hh:262
(inlined by)seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<row_level_repair::run()::{lambda()#1}>::type>::type seastar::async<row_level_repair::run()::{lambda()#1}>(seastar::thread_attributes, std::invoke_result&&, (row_level_repair::run()::{lambda()#1}&&)...)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at /tmp/./seastar/include/seastar/util/noncopyable_function.hh:129
seastar::noncopyable_function<void ()>::operator()() const at ./build/release/seastar/./seastar/include/seastar/util/noncopyable_function.hh:215
(inlined by)seastar::thread_context::main() at ./build/release/seastar/./seastar/src/core/thread.cc:311
Is it caused by a memory allocation failure during repair?
scylla[190229]: [shard 19:stre] repair - repair[345e0a07-7015-461b-8d93-918f1444e601]: shard=19, keyspace=alternator_test, cf=test, range=(-998202281619702246,-997789613637636054], got error in row level repair: std::bad_alloc (std::bad_alloc)
Are you reporting the same as The expansion of the 180-node cluster has failed by any chance? Heavy memory pressure and both in ScyllaDB 5.4. 
Is it caused by a memory allocation failure during repair?
The decoded backtrace definitely points to row_level repair codepath. You may disable it and fallback to old streaming and see if it helps. But as in the aforementioned forum post, do check how your LSA/Non-LSA consumptions were at the time of the crash and take it from there.
Same upgrade recommendations and next steps apply.