Matt_J
November 29, 2023, 9:24am
1
Hey, I’m having hard time starting 3 nodes setup in docker on my local env.
3rd node is reporting
FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)
2023-11-28 12:55:33,990 INFO exited: scylla (exit status 7; not expected)
2023-11-28 12:55:34,991 INFO gave up: scylla entered FATAL state, too many start retries too quickly
The image is scylladb/scylla:5.2.0 and host machine is mac osx with intel processor. Any idea how to set this aio parameter on mac ?
Try running sysctl -w kern.aiomax=1048576
Matt_J
December 1, 2023, 6:16am
3
➜ ~ sudo sysctl -w kern.aiomax=1048576 Password: kern.aiomax: 90 sysctl: kern.aiomax=1048576: Invalid argument ➜ ~ sudo sysctl -w kern.aiomax="1048576" kern.aiomax: 90 sysctl: kern.aiomax=1048576: Invalid argument
on macos it looks like -w is not valid
I can set like this but the highest value it would let me set is 11000 (12000 and above said invalid)
sudo sysctl kern.aiomax=11000
so stuck on the same thing. I might just try linux but would be nice if containers worked on macos
roy
January 18, 2024, 10:00pm
7
On a intel based chip with “2.8 GHz Quad-Core Intel Core i7” I’m able to start a 3 node cluster, each with --smp 1.
$ sysctl -n hw.ncpu
8
sysctl -a | grep aio
kern.aiomax: 90
kern.aioprocmax: 16
kern.aiothreads: 4
$ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.17.0.3 572 KB 256 ? 0028d9e0-e771-45eb-b922-f15660b7b3a8 rack1
UN 172.17.0.2 500 KB 256 ? 06a0db13-20a1-43a8-a7ff-c60c0128066b rack1
UN 172.17.0.4 696 KB 256 ? f136cc48-1b17-4626-b584-d2898d87f8f1 rack1
However, on a apple chip based (Apple M3 Pro), 2 nodes starts ok with --smp 1, but the 3 node has the issue of max-aio-nr.
sysctl -n hw.ncpu
11
sysctl -a | grep aio
kern.aiomax: 90
kern.aioprocmax: 16
kern.aiothreads: 4
1 Like
roy
January 25, 2024, 1:59pm
8
I’m happy to share that there is an easy workaround that works also for macOS:
You can start scylla with the following flag: “–reactor-backend=epoll ”
For Example:
docker run --name some-scylla --hostname some-scylla -d scylladb/scylla --reactor-backend=epoll
docker run --name some-scylla2 --hostname some-scylla2 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"
docker run --name some-scylla3 --hostname some-scylla3 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"
15:53 roy@macbook-pro ~ () $ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN 172.17.0.3 369.39 KB 256 ? dd207e02-468e-48c2-91ca-336cafff8f2f rack1
UN 172.17.0.2 186.78 KB 256 ? fa67ef85-468d-4c7c-8275-46bc3a6931e5 rack1
UN 172.17.0.4 566.11 KB 256 ? 444b5568-4500-410b-b1b0-2676c4f908cc rack1