Scylladb entering into a crashloop

ScyllaDB
1

I tried to setup scylladb in a docker container using compose using this yml for docker compose

version: '3'
services:
    scylladb:
      image: scylladb/scylla
      ports:
        - "9042:9042"
      volumes:
        - ./dev/scylla_data:/var/lib/scylla

it keeps spitting errors out like it’s no tommorow

cinnamon_blog_v3-scylladb-1  | ERROR 2023-11-22 14:58:29,474 [shard 1] table - failed to write sstable /var/lib/scylla/data/system/local-7ad54392bcdd35a684174e047860b377/mc-5-big-Data.db: std::system_error (No such file or directory)
cinnamon_blog_v3-scylladb-1  | ERROR 2023-11-22 14:58:29,475 [shard 1] table - Memtable flush failed due to: std::system_error (error system:2, No such file or directory). Aborting, at 0x539e1be 0x539e770 0x539ea58 0x19754d6 0x1201cfa 0x50106a4 0x5011927 0x5032711 0x4fe2b7a /opt/scylladb/libreloc/libc.so.6+0x8b12c /opt/scylladb/libreloc/libc.so.6+0x10cbbf
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::internal::coroutine_traits_base<void>::promise_type
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::internal::coroutine_traits_base<void>::promise_type
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}>(seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false> >(seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(auto:1) const::{lambda()#1}, false>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
cinnamon_blog_v3-scylladb-1  |    --------

 --------
cinnamon_blog_v3-scylladb-1  |    seastar::internal::coroutine_traits_base<void>::promise_type
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::internal::coroutine_traits_base<void>::promise_type
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}>(seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6&&)::{lambda(auto:1&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::handle_exception<replica::dirty_memory_manager::flush_one(replica::memtable_list&, replica::flush_permit&&)::$_6>(auto:1&&)::{lambda(auto:1&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false> >(seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(replica::flush_permit) const::{lambda()#1}, false>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::finally_body<replica::memtable_list::flush()::$_32::operator()<replica::flush_permit>(auto:1) const::{lambda()#1}, false>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
cinnamon_blog_v3-scylladb-1  |    --------
cinnamon_blog_v3-scylladb-1  |    seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}, seastar::future<void>::then_wrapped_nrvo<void, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}>(seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::shared_future<>::shared_state::get_future(std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda(seastar::future<void>&&)#1}&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>
cinnamon_blog_v3-scylladb-1  | Aborting on shard 1.
cinnamon_blog_v3-scylladb-1  | Backtrace:
cinnamon_blog_v3-scylladb-1  |   0x4fffe28
cinnamon_blog_v3-scylladb-1  |   0x5032262
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x3cb1f
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x8ce5b
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x3ca75
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x267fb
cinnamon_blog_v3-scylladb-1  |   0x1975511
cinnamon_blog_v3-scylladb-1  |   0x1201cfa
cinnamon_blog_v3-scylladb-1  |   0x50106a4
cinnamon_blog_v3-scylladb-1  |   0x5011927
cinnamon_blog_v3-scylladb-1  |   0x5032711
cinnamon_blog_v3-scylladb-1  |   0x4fe2b7a
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x8b12c
cinnamon_blog_v3-scylladb-1  |   /opt/scylladb/libreloc/libc.so.6+0x10cbbf
cinnamon_blog_v3-scylladb-1  | 2023-11-22 14:58:29,591 INFO exited: scylla (terminated by SIGABRT; not expected)
cinnamon_blog_v3-scylladb-1  | 2023-11-22 14:58:30,594 INFO spawned: 'scylla' with pid 185
cinnamon_blog_v3-scylladb-1  | Scylla version 5.2.10-0.20231102.454e5a711099 with build-id 1cabeff053229b347a86aefb619c61419e5b647e starting ...
cinnamon_blog_v3-scylladb-1  | command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --overprovisioned --listen-address 172.22.0.2 --rpc-address 172.22.0.2 --seed-provider-parameters seeds=172.22.0.2 --alternator-address 172.22.0.2 --blocked-reactor-notify-ms 999999999"

please help!

EDIT:

this is running docker compose on windows with docker running on a virtual machine using WSL

2

Is ./dev/scylla_data within WSL’s 9P filesystem protocol (ie: the /mnt/c mapping?), or is it within the virtual SDD file (default ext4.vhdx)?

I don’t have a Windows machine available atm, but I would start by trying to start the container without the volume. It seems to be the cause of your problem.

If 9P is what you are using, then it is simple: It isn’t POSIX compliant. If it isn’t, then it is likely something else I couldn’t figure just from these snippets.

3

Fair enough, I tried on my work machine, and it seems to be okay with that volume on linux, grr why must windows be like this??