Allocation problems 6.1

ScyllaDB
,
1

Hi,

ScyllaDB version 6.1.4 seems to have less memory allocation warnings than 6.0.

Nevertheless we are sporadically still seeing some allocator failures:

Dec 30 08:44:14 o-p-L309-7 scylla[1843673]:  [shard 0:stmt] lsa - Standard allocator failure, increasing head-room in section 0x60000879c680 to 134217728 [B]; trace:
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:97
seastar::current_tasktrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:148
seastar::current_backtrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:181
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ././utils/logalloc.cc:3015
cache::cache_mutation_reader::do_fill_buffer() at ././utils/logalloc.hh:510
cache::cache_mutation_reader::fill_buffer()::{lambda()#3}::operator()() const at ././cache_mutation_reader.hh:324
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&>(cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:2032
 (inlined by) auto seastar::futurize_invoke<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&>(cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:2063
 (inlined by) seastar::future<void> seastar::do_until<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}, cache::cache_mutation_reader::fill_buffer()::{lambda()#2}>(cache::cache_mutation_reader::fill_buffer()::{lambda()#2}, cache::cache_mutation_reader::fill_buffer()::{lambda()#3}) requires (std::is_invocable_r_v<bool, cache::cache_mutation_reader::fill_buffer()::{lambda()#2}>)&&(std::is_invocable_r_v<seastar::future<void>, cache::cache_mutation_reader::fill_buffer()::{lambda()#3}>) at ././seastar/include/seastar/core/loop.hh:349
 (inlined by) cache::cache_mutation_reader::fill_buffer() at ././cache_mutation_reader.hh:323
cache::cache_mutation_reader::fill_buffer() at ././cache_mutation_reader.hh:314
mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}::operator()(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) const at ././readers/mutation_reader.hh:211
seastar::future<void> std::__invoke_impl<seastar::future<void>, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>(std::__invoke_other, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:61
 (inlined by) std::__invoke_result<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>::type std::__invoke<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>(mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96
 (inlined by) decltype(auto) std::__apply_impl<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, std::tuple<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >&, 0ul>(mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, std::tuple<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >&, std::integer_sequence<unsigned long, 0ul>) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921
 (inlined by) _ZSt5applyIRZN15mutation_reader4impl7consumeI20compact_for_query_v2I20query_result_builderEQ19FlattenedConsumerV2IT_EEEDaS6_EUlRNS1_16consumer_adapterIS5_EEE_TkSt12__tuple_likeRSt5tupleIJS8_EEEDcOS6_OT0_ at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2936
 (inlined by) auto seastar::internal::do_with_impl<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/do_with.hh:99
 (inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/future.hh:2032
 (inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/future.hh:2063
 (inlined by) auto seastar::do_with<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/do_with.hh:137
query::consume_page<query_result_builder>(mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) requires CompactedFragmentsConsumerV2<query_result_builder>::{lambda(mutation_fragment_v2*)#1}::operator()(mutation_fragment_v2*) at ././readers/mutation_reader.hh:323
auto query::consume_page<query_result_builder>(mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) requires CompactedFragmentsConsumerV2<query_result_builder> at ././seastar/include/seastar/core/future.hh:2032
auto query::querier::consume_page<query_result_builder>(query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >, tracing::trace_state_ptr) requires CompactedFragmentsConsumerV2<query_result_builder> at ././querier.hh:173
replica::table::query(seastar::lw_shared_ptr<schema const>, reader_permit, query::read_command const&, query::result_options, std::vector<interval<dht::ring_position>, std::allocator<interval<dht::ring_position> > > const&, tracing::trace_state_ptr, query::result_memory_limiter&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::optional<query::querier>*) at ././replica/table.cc:3261
seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<interval<dht::ring_position>, std::allocator<interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_0>::call(seastar::noncopyable_function<seastar::future<void> (reader_permit)> const*, reader_permit) at ././replica/database.cc:1543
reader_concurrency_semaphore::execution_loop() at ././seastar/include/seastar/util/noncopyable_function.hh:215
std::__n4861::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume() const at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/coroutine:242
 (inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at ././seastar/include/seastar/core/coroutine.hh:122
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:2651
seastar::reactor::run_some_tasks() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3114
seastar::reactor::do_run() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3282
seastar::reactor::run() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3172
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/app-template.cc:167
scylla_main(int, char**) at ././main.cc:699
std::function<int (int, char**)>::operator()(int, char**) const at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/std_function.h:591
main at ././main.cc:2208
/data/scylla-s3-reloc.cache/by-build-id/5c8468a0d5595faaed925ba3e164c9e68de3c0e6/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8f53abaad945a669f2bdcd25f471d80e077568ef, for GNU/Linux 3.2.0, not stripped

__libc_start_call_main at ??:?
__libc_start_main_alias_2 at :?
_start at ??:?

An allocation of 128MB seems rather large. Might this be an issue with scylla or are we doing something wrong in our queries?

  • We basically do not use collections.
  • There are quite some large cells up to 3MB. But it cannot go larger than that as our application checks for that and throws exception during write.
  • No large rows reported.
  • Partitions: some go up to 1GB and 30m rows. I would hope that this is not a problem.
  • Some queries might be rather large
2

Today I saw a strange one, related to compactions:

Jan 01 14:59:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 f5715980-c850-11ef-a5a5-881419ad6d25] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_0qsw_0lv1c2ionkwn0o5gyt-big-Data.db:level=0:origin=compaction]
Jan 01 14:59:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 f5715980-c850-11ef-a5a5-881419ad6d25] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:97
seastar::current_tasktrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:148
seastar::current_backtrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:181
sstables::compaction::on_interrupt(std::__exception_ptr::exception_ptr) at ././compaction/compaction.cc:872
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././compaction/compaction.cc:1741
seastar::noncopyable_function<void ()>::operator()() const at ./build/release/seastar/./seastar/include/seastar/util/noncopyable_function.hh:215
 (inlined by) seastar::thread_context::main() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/thread.cc:311

Jan 01 15:04:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 a84205a0-c851-11ef-a5a5-881419ad6d25] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_0qsw_0lv1c2ionkwn0o5gyt-big-Data.db:level=0:origin=compaction]
Jan 01 15:04:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 a84205a0-c851-11ef-a5a5-881419ad6d25] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0] Already seen, not resolving again.

Jan 01 15:09:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 5b12b1c0-c852-11ef-a5a5-881419ad6d25] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_0qsw_0lv1c2ionkwn0o5gyt-big-Data.db:level=0:origin=compaction]
Jan 01 15:09:11 o-p-L309-7 scylla[1843673]:  [shard 0:comp] compaction - [Compact pc.tntindex2 5b12b1c0-c852-11ef-a5a5-881419ad6d25] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0] Already seen, not resolving again.





Jan 01 14:54:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 4a330050-c850-11ef-9b72-01dccbaf0ec1] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_11ow_5h6rk2d3lp8pfwrr9d-big-Data.db:level=0:origin=compaction]
Jan 01 14:54:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 4a330050-c850-11ef-9b72-01dccbaf0ec1] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:97
seastar::current_tasktrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:148
seastar::current_backtrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:181
sstables::compaction::on_interrupt(std::__exception_ptr::exception_ptr) at ././compaction/compaction.cc:872
seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::async<sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0>(seastar::thread_attributes, sstables::compaction::run(std::unique_ptr<sstables::compaction, std::default_delete<sstables::compaction> >)::$_0&&)::{lambda()#1}>::call(seastar::noncopyable_function<void ()> const*) at ././compaction/compaction.cc:1741
seastar::noncopyable_function<void ()>::operator()() const at ./build/release/seastar/./seastar/include/seastar/util/noncopyable_function.hh:215
 (inlined by) seastar::thread_context::main() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/thread.cc:311

Jan 01 14:59:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 fd03ac70-c850-11ef-9b72-01dccbaf0ec1] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_11ow_5h6rk2d3lp8pfwrr9d-big-Data.db:level=0:origin=compaction]
Jan 01 14:59:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 fd03ac70-c850-11ef-9b72-01dccbaf0ec1] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0] Already seen, not resolving again.

Jan 01 15:04:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 afd45890-c851-11ef-9b72-01dccbaf0ec1] Compacting [/data/scylla/data/pc/tntindex2-9a443dd0a31011e9b8d6000000000000/me-3gmj_11ow_5h6rk2d3lp8pfwrr9d-big-Data.db:level=0:origin=compaction]
Jan 01 15:04:23 o-p-L309-5 scylla[1823816]:  [shard 4:comp] compaction - [Compact pc.tntindex2 afd45890-c851-11ef-9b72-01dccbaf0ec1] Compacting of 1 sstables interrupted due to: std::bad_alloc (std::bad_alloc), at
[Backtrace #0] Already seen, not resolving again.

A alloc error during compaction seems worrisome to me.

3

Do these mean that its loading 128MB of mutations in a query?

N7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE21handle_exception_typeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS4_INS7_5tupleIJNS_11foreign_ptrINS_13lw_shared_ptrIN5query6resultEEEEE17cache_temperatureN7replica17exception_variantEEEEEERKNS7_11client_infoENS7_14opt_time_pointENSF_12read_commandE17wrapping_intervalIN3dht13ring_positionEENS7_8optionalINSF_16digest_algorithmEEENSX_ISt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS13_19account_and_enforceEEEEENSX_IN7service13fencing_tokenEEEEESN_JSS_SW_SZ_S17_S1A_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESt8optionalINSt6chrono10time_pointINS_12lowres_clockENS1U_8durationIlSt5ratioILl1ELl1000000000EEEEEEElNS7_7rcv_bufENS_4gate6holderEE_clES1S_S22_lS23_S25_ENUlT_E_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1W_EEEEDaS27_EUlRNS_21gate_closed_exceptionEE_EES5_OS27_EUlS2G_E_ZNS5_17then_wrapped_nrvoIS5_S2H_EENS_8futurizeIS27_E4typeES1M_EUlOS3_RS2H_ONS_12future_stateINS1_9monostateEEEE_vEE
Jan 09 17:04:27 o-p-L309-1 scylla[47422]:  [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603008ad4680 to 16777216 [B]; trace:
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
 (inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:97
seastar::current_tasktrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:148
seastar::current_backtrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:181
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ././utils/logalloc.cc:3015
cache::cache_mutation_reader::do_fill_buffer() at ././utils/logalloc.hh:510
cache::cache_mutation_reader::fill_buffer()::{lambda()#3}::operator()() const at ././cache_mutation_reader.hh:324
 (inlined by) seastar::internal::do_until_state<cache::cache_mutation_reader::fill_buffer()::{lambda()#2}, cache::cache_mutation_reader::fill_buffer()::{lambda()#3}>::run_and_dispose() at ././seastar/include/seastar/core/loop.hh:303
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:2651
seastar::reactor::run_some_tasks() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3114
seastar::reactor::do_run() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3282
std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::$_0>::_M_invoke(std::_Any_data const&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:4525
seastar::posix_thread::start_routine(void*) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/std_function.h:591
/data/scylla-s3-reloc.cache/by-build-id/5c8468a0d5595faaed925ba3e164c9e68de3c0e6/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8f53abaad945a669f2bdcd25f471d80e077568ef, for GNU/Linux 3.2.0, not stripped

start_thread at ??:?
__clone3 at :?

N7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE21handle_exception_typeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS4_INS7_5tupleIJNS_11foreign_ptrINS_13lw_shared_ptrIN5query6resultEEEEE17cache_temperatureN7replica17exception_variantEEEEEERKNS7_11client_infoENS7_14opt_time_pointENSF_12read_commandE17wrapping_intervalIN3dht13ring_positionEENS7_8optionalINSF_16digest_algorithmEEENSX_ISt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS13_19account_and_enforceEEEEENSX_IN7service13fencing_tokenEEEEESN_JSS_SW_SZ_S17_S1A_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESt8optionalINSt6chrono10time_pointINS_12lowres_clockENS1U_8durationIlSt5ratioILl1ELl1000000000EEEEEEElNS7_7rcv_bufENS_4gate6holderEE_clES1S_S22_lS23_S25_ENUlT_E_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1W_EEEEDaS27_EUlRNS_21gate_closed_exceptionEE_EES5_OS27_EUlS2G_E_ZNS5_17then_wrapped_nrvoIS5_S2H_EENS_8futurizeIS27_E4typeES1M_EUlOS3_RS2H_ONS_12future_stateINS1_9monostateEEEE_vEE
Jan 09 17:04:27 o-p-L309-1 scylla[47422]:  [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603008ad4680 to 33554432 [B]; trace:
[Backtrace #0] Already seen, not resolving again.

N7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE21handle_exception_typeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS4_INS7_5tupleIJNS_11foreign_ptrINS_13lw_shared_ptrIN5query6resultEEEEE17cache_temperatureN7replica17exception_variantEEEEEERKNS7_11client_infoENS7_14opt_time_pointENSF_12read_commandE17wrapping_intervalIN3dht13ring_positionEENS7_8optionalINSF_16digest_algorithmEEENSX_ISt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS13_19account_and_enforceEEEEENSX_IN7service13fencing_tokenEEEEESN_JSS_SW_SZ_S17_S1A_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESt8optionalINSt6chrono10time_pointINS_12lowres_clockENS1U_8durationIlSt5ratioILl1ELl1000000000EEEEEEElNS7_7rcv_bufENS_4gate6holderEE_clES1S_S22_lS23_S25_ENUlT_E_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1W_EEEEDaS27_EUlRNS_21gate_closed_exceptionEE_EES5_OS27_EUlS2G_E_ZNS5_17then_wrapped_nrvoIS5_S2H_EENS_8futurizeIS27_E4typeES1M_EUlOS3_RS2H_ONS_12future_stateINS1_9monostateEEEE_vEE
Jan 09 17:04:27 o-p-L309-1 scylla[47422]:  [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603008ad4680 to 67108864 [B]; trace:
[Backtrace #0] Already seen, not resolving again.

N7seastar12continuationINS_8internal22promise_base_with_typeIvEEZNS_6futureIvE21handle_exception_typeIZZZNS_3rpc11recv_helperIN4netw10serializerESt8functionIFNS4_INS7_5tupleIJNS_11foreign_ptrINS_13lw_shared_ptrIN5query6resultEEEEE17cache_temperatureN7replica17exception_variantEEEEEERKNS7_11client_infoENS7_14opt_time_pointENSF_12read_commandE17wrapping_intervalIN3dht13ring_positionEENS7_8optionalINSF_16digest_algorithmEEENSX_ISt7variantIJSt9monostateN2db24per_partition_rate_limit12account_onlyENS13_19account_and_enforceEEEEENSX_IN7service13fencing_tokenEEEEESN_JSS_SW_SZ_S17_S1A_ENS7_19do_want_client_infoENS7_18do_want_time_pointEEEDaNS7_9signatureIFT1_DpT2_EEEOT0_T3_T4_ENUlNS_10shared_ptrINS7_6server10connectionEEESt8optionalINSt6chrono10time_pointINS_12lowres_clockENS1U_8durationIlSt5ratioILl1ELl1000000000EEEEEEElNS7_7rcv_bufENS_4gate6holderEE_clES1S_S22_lS23_S25_ENUlT_E_clINS_15semaphore_unitsINS_35semaphore_default_exception_factoryES1W_EEEEDaS27_EUlRNS_21gate_closed_exceptionEE_EES5_OS27_EUlS2G_E_ZNS5_17then_wrapped_nrvoIS5_S2H_EENS_8futurizeIS27_E4typeES1M_EUlOS3_RS2H_ONS_12future_stateINS1_9monostateEEEE_vEE
Jan 09 17:04:27 o-p-L309-1 scylla[47422]:  [shard 3:stmt] lsa - Standard allocator failure, increasing head-room in section 0x603008ad4680 to 134217728 [B]; trace:
[Backtrace #0] Already seen, not resolving again.
4

The allocation warnings and failures you’re seeing can be due to large partitions and large cells, as well as internal memory management details of Scylla’s allocator. Here are some points to consider:

  • Allocations of 128MB or more can happen if a single partition is very large (you mentioned some partitions up to 1GB with 30 million rows), which puts high pressure on memory allocation and can trigger allocator headroom increases or failures.

  • Large cells (up to 3MB) are generally acceptable but combined with large partitions can exacerbate memory pressure.

  • Scylla’s internal allocator (logalloc) may increase allocation granularity under load, which is normal but can cause warnings during peak demand.

  • Frequent memory allocation failures might also correlate with query patterns or long-running queries that require large buffers.

  • It is recommended to review data model to avoid extremely large partitions if possible (e.g., split data into smaller partitions).

  • Monitoring and tuning JVM heap, page cache, and memory-related settings in Scylla may help, but the fundamental issue here is the workload profile and partition size.

  • If patterns of frequent bad_alloc or allocator failures persist and impact stability, consider reaching out to ScyllaDB support with detailed logs and metrics to investigate potential improvements or fixes.

In summary, this is likely not a bug but a symptom of heavy memory pressure caused by very large partitions and data sizes, combined with high query demands. The best path is to monitor partition sizes, avoid extremely large partitions if possible, and tune memory settings accordingly.

I would also recommend upgrading to a recent version - ScyllaDB 2025.3.0 and see if the issues persist.

5

Thanks for the feedback!

sorry, I forgot to update the thread:

Since we have upgraded to 2024.1.8 our issues are gone and the logs are also so much cleaner (due to `managed_bytes` violates preferred contiguous allocation size · Issue #23781 · scylladb/scylladb · GitHub).

I think these few large partitions we have are basically never read and therefore dont cause any/much harm. As since that fix our logs are pretty clean. Nevertheless we are trying to remove the large partitions.

1 Like