Hi,
ScyllaDB version 6.1.4 seems to have less memory allocation warnings than 6.0.
Nevertheless we are sporadically still seeing some allocator failures:
Dec 30 08:44:14 o-p-L309-7 scylla[1843673]: [shard 0:stmt] lsa - Standard allocator failure, increasing head-room in section 0x60000879c680 to 134217728 [B]; trace:
[Backtrace #0]
void seastar::backtrace<seastar::current_backtrace_tasklocal()::$_0>(seastar::current_backtrace_tasklocal()::$_0&&) at ./build/release/seastar/./seastar/include/seastar/util/backtrace.hh:68
(inlined by) seastar::current_backtrace_tasklocal() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:97
seastar::current_tasktrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:148
seastar::current_backtrace() at ./build/release/seastar/./build/release/seastar/./seastar/src/util/backtrace.cc:181
logalloc::allocating_section::on_alloc_failure(logalloc::region&) at ././utils/logalloc.cc:3015
cache::cache_mutation_reader::do_fill_buffer() at ././utils/logalloc.hh:510
cache::cache_mutation_reader::fill_buffer()::{lambda()#3}::operator()() const at ././cache_mutation_reader.hh:324
(inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&>(cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:2032
(inlined by) auto seastar::futurize_invoke<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&>(cache::cache_mutation_reader::fill_buffer()::{lambda()#3}&) at ././seastar/include/seastar/core/future.hh:2063
(inlined by) seastar::future<void> seastar::do_until<cache::cache_mutation_reader::fill_buffer()::{lambda()#3}, cache::cache_mutation_reader::fill_buffer()::{lambda()#2}>(cache::cache_mutation_reader::fill_buffer()::{lambda()#2}, cache::cache_mutation_reader::fill_buffer()::{lambda()#3}) requires (std::is_invocable_r_v<bool, cache::cache_mutation_reader::fill_buffer()::{lambda()#2}>)&&(std::is_invocable_r_v<seastar::future<void>, cache::cache_mutation_reader::fill_buffer()::{lambda()#3}>) at ././seastar/include/seastar/core/loop.hh:349
(inlined by) cache::cache_mutation_reader::fill_buffer() at ././cache_mutation_reader.hh:323
cache::cache_mutation_reader::fill_buffer() at ././cache_mutation_reader.hh:314
mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}::operator()(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) const at ././readers/mutation_reader.hh:211
seastar::future<void> std::__invoke_impl<seastar::future<void>, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>(std::__invoke_other, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:61
(inlined by) std::__invoke_result<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>::type std::__invoke<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&>(mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/invoke.h:96
(inlined by) decltype(auto) std::__apply_impl<mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, std::tuple<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >&, 0ul>(mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&, std::tuple<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> > >&, std::integer_sequence<unsigned long, 0ul>) at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2921
(inlined by) _ZSt5applyIRZN15mutation_reader4impl7consumeI20compact_for_query_v2I20query_result_builderEQ19FlattenedConsumerV2IT_EEEDaS6_EUlRNS1_16consumer_adapterIS5_EEE_TkSt12__tuple_likeRSt5tupleIJS8_EEEDcOS6_OT0_ at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/tuple:2936
(inlined by) auto seastar::internal::do_with_impl<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/do_with.hh:99
(inlined by) seastar::future<void> seastar::futurize<seastar::future<void> >::invoke<seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/future.hh:2032
(inlined by) auto seastar::futurize_invoke<seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(seastar::future<void> (*&)(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&), mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/future.hh:2063
(inlined by) auto seastar::do_with<mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}>(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&&, mutation_reader::impl::consume<compact_for_query_v2<query_result_builder> >(compact_for_query_v2<query_result_builder>) requires FlattenedConsumerV2<compact_for_query_v2<query_result_builder> >::{lambda(mutation_reader::impl::consumer_adapter<compact_for_query_v2<query_result_builder> >&)#1}&&) at ././seastar/include/seastar/core/do_with.hh:137
query::consume_page<query_result_builder>(mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) requires CompactedFragmentsConsumerV2<query_result_builder>::{lambda(mutation_fragment_v2*)#1}::operator()(mutation_fragment_v2*) at ././readers/mutation_reader.hh:323
auto query::consume_page<query_result_builder>(mutation_reader&, seastar::lw_shared_ptr<compact_mutation_state<(compact_for_sstables)0> >, query::partition_slice const&, query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >) requires CompactedFragmentsConsumerV2<query_result_builder> at ././seastar/include/seastar/core/future.hh:2032
auto query::querier::consume_page<query_result_builder>(query_result_builder&&, unsigned long, unsigned int, std::chrono::time_point<gc_clock, std::chrono::duration<long, std::ratio<1l, 1l> > >, tracing::trace_state_ptr) requires CompactedFragmentsConsumerV2<query_result_builder> at ././querier.hh:173
replica::table::query(seastar::lw_shared_ptr<schema const>, reader_permit, query::read_command const&, query::result_options, std::vector<interval<dht::ring_position>, std::allocator<interval<dht::ring_position> > > const&, tracing::trace_state_ptr, query::result_memory_limiter&, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::optional<query::querier>*) at ././replica/table.cc:3261
seastar::noncopyable_function<seastar::future<void> (reader_permit)>::indirect_vtable_for<replica::database::query(seastar::lw_shared_ptr<schema const>, query::read_command const&, query::result_options, std::vector<interval<dht::ring_position>, std::allocator<interval<dht::ring_position> > > const&, tracing::trace_state_ptr, std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >, std::variant<std::monostate, db::per_partition_rate_limit::account_only, db::per_partition_rate_limit::account_and_enforce>)::$_0>::call(seastar::noncopyable_function<seastar::future<void> (reader_permit)> const*, reader_permit) at ././replica/database.cc:1543
reader_concurrency_semaphore::execution_loop() at ././seastar/include/seastar/util/noncopyable_function.hh:215
std::__n4861::coroutine_handle<seastar::internal::coroutine_traits_base<void>::promise_type>::resume() const at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/coroutine:242
(inlined by) seastar::internal::coroutine_traits_base<void>::promise_type::run_and_dispose() at ././seastar/include/seastar/core/coroutine.hh:122
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:2651
seastar::reactor::run_some_tasks() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3114
seastar::reactor::do_run() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3282
seastar::reactor::run() at ./build/release/seastar/./build/release/seastar/./seastar/src/core/reactor.cc:3172
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/app-template.cc:276
seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at ./build/release/seastar/./build/release/seastar/./seastar/src/core/app-template.cc:167
scylla_main(int, char**) at ././main.cc:699
std::function<int (int, char**)>::operator()(int, char**) const at /usr/lib/gcc/x86_64-redhat-linux/14/../../../../include/c++/14/bits/std_function.h:591
main at ././main.cc:2208
/data/scylla-s3-reloc.cache/by-build-id/5c8468a0d5595faaed925ba3e164c9e68de3c0e6/extracted/scylla/libreloc/libc.so.6: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=8f53abaad945a669f2bdcd25f471d80e077568ef, for GNU/Linux 3.2.0, not stripped
__libc_start_call_main at ??:?
__libc_start_main_alias_2 at :?
_start at ??:?
An allocation of 128MB seems rather large. Might this be an issue with scylla or are we doing something wrong in our queries?
- We basically do not use collections.
- There are quite some large cells up to 3MB. But it cannot go larger than that as our application checks for that and throws exception during write.
- No large rows reported.
- Partitions: some go up to 1GB and 30m rows. I would hope that this is not a problem.
- Some queries might be rather large