Running 3 node scylladb in docker

ScyllaDB
1

Hey, I’m having hard time starting 3 nodes setup in docker on my local env.
3rd node is reporting

FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)
2023-11-28 12:55:33,990 INFO exited: scylla (exit status 7; not expected)
2023-11-28 12:55:34,991 INFO gave up: scylla entered FATAL state, too many start retries too quickly

The image is scylladb/scylla:5.2.0 and host machine is mac osx with intel processor. Any idea how to set this aio parameter on mac ?

2

Try running sysctl -w kern.aiomax=1048576

3

➜ ~ sudo sysctl -w kern.aiomax=1048576 Password: kern.aiomax: 90 sysctl: kern.aiomax=1048576: Invalid argument ➜ ~ sudo sysctl -w kern.aiomax="1048576" kern.aiomax: 90 sysctl: kern.aiomax=1048576: Invalid argument

4

on macos it looks like -w is not valid
I can set like this but the highest value it would let me set is 11000 (12000 and above said invalid)

sudo sysctl kern.aiomax=11000

so stuck on the same thing. I might just try linux but would be nice if containers worked on macos

6

I created a bug report about this problem: Docker on macOS fails with: "Could not setup Async I/O: Resource temporarily unavailable." · Issue #16806 · scylladb/scylladb · GitHub

7

On a intel based chip with “2.8 GHz Quad-Core Intel Core i7” I’m able to start a 3 node cluster, each with --smp 1.

$ sysctl -n hw.ncpu
8

sysctl -a | grep aio
kern.aiomax: 90
kern.aioprocmax: 16
kern.aiothreads: 4

 $ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.3  572 KB     256          ?       0028d9e0-e771-45eb-b922-f15660b7b3a8  rack1
UN  172.17.0.2  500 KB     256          ?       06a0db13-20a1-43a8-a7ff-c60c0128066b  rack1
UN  172.17.0.4  696 KB     256          ?       f136cc48-1b17-4626-b584-d2898d87f8f1  rack1

However, on a apple chip based (Apple M3 Pro), 2 nodes starts ok with --smp 1, but the 3 node has the issue of max-aio-nr.

sysctl -n hw.ncpu
11

sysctl -a | grep aio
kern.aiomax: 90
kern.aioprocmax: 16
kern.aiothreads: 4

1 Like
8

I’m happy to share that there is an easy workaround that works also for macOS:

You can start scylla with the following flag: “–reactor-backend=epoll”

For Example:

docker run --name some-scylla --hostname some-scylla -d scylladb/scylla --reactor-backend=epoll
docker run --name some-scylla2  --hostname some-scylla2 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"
docker run --name some-scylla3  --hostname some-scylla3 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"

15:53 roy@macbook-pro ~ () $ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.3  369.39 KB  256          ?       dd207e02-468e-48c2-91ca-336cafff8f2f  rack1
UN  172.17.0.2  186.78 KB  256          ?       fa67ef85-468d-4c7c-8275-46bc3a6931e5  rack1
UN  172.17.0.4  566.11 KB  256          ?       444b5568-4500-410b-b1b0-2676c4f908cc  rack1