Ubuntu Linux - Error starting 3-node ScyllaDB on Docker

gjjohnson · January 15, 2024, 7:05pm

I am trying to create a 3-node ScyllaDB using Docker running on Ubuntu Linux 22.04. I am executing docker run commands as a non-root user.

This is from the S101 Scylla DB Essentials Lab Overview and Setup instructions. I have set fs.aio-max-nr to 1048576 in /etc/sysctl.conf and have run sysctl -p /etc/sysctl.conf.

root@M93p:/etc# sysctl -p /etc/sysctl.conf
fs.aio-max-nr = 1048576

The lab provides docker run syntax to create three nodes (Node_X, Node_Y and Node_Z). I am able to create Node_X and Node_Y, but when creating Node_Z, I get the following error:

FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)

lscpu on my workstation shows:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Vendor ID: GenuineIntel
Model name: Intel(R) Core™ i5-4570 CPU @ 3.20GHz
CPU family: 6
Model: 60
Thread(s) per core: 1
Core(s) per socket: 4

The workstation has 24GB of memory and I am taking the defaults on Docker for the three containers.

Any suggestions on a work-around for this issue?

Guy · January 16, 2024, 5:56am

Thanks for reporting, please send the logs.

gjjohnson · January 17, 2024, 7:26am

I am executing a docker run to create three containers for ScyllaDB Nodes: Node_X, Node_Y and Node_Z:

I have set fs.aio-max-nr to double the recommended size – this is running on a Intel 4-core workstation with 24GB memory.

gjjohnson@M93p:~$ sudo sysctl -w fs.aio-max-nr=2560000
fs.aio-max-nr = 2560000

I have set this kernel parameter in /etc/sysctl.conf and have run a sysctl -p
sudo sysctl -p /etc/sysctl.conf

Node_X and Node_Y startup successfully with the following message:
init - Scylla version 5.2.0-0.20230427.429b696bbc1b initialization completed.
I wait for an initialization complete message in Node_X before running Node_Y and then wait for Node_Y to complete initialization before running Node_Z.

When I attempt to start Node_Z it fails with the following message:
FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)

Since I get the same message after increasing fs.aio-max-nr, I decided to try to limit each container to one CPU.
Again, Node_X and Node_Y start successfully and Node_Z fails with the same error.

gjjohnson · January 17, 2024, 7:26am

Let me know if you also need the logs from Node_X and Node_Y.

roy · January 18, 2024, 11:12pm

Recently it was reported the same also for macOS here - Docker on macOS fails with: "Could not setup Async I/O: Resource temporarily unavailable." · Issue #16806 · scylladb/scylladb · GitHub.

However, if it happens also in other platforms it may get higher priority.
Can you please add your input in that issue as well?

gjjohnson · January 19, 2024, 1:32am

Yes, I will.

I read the MacOS issue and it is the same error, but the suggested Kernel parameters mentioned for the MacOS are different than the Ubuntu Kernel parameters.

gjjohnson · January 24, 2024, 9:16am

Ok – I used cloud.scylladb.com to run the exercises in S101: ScyllaDB Essentials. I’m now working on S110: The Mutant Monitoring System and the lab setup is using docker-compose to create a local cluster. I’ve removed all of my previous containers, run git clone to download all of the required configuration files and code samples, and have re-started Docker desktop and have run docker-compose up -d. I am still encountering the same error starting one of the three nodes in the cluster.

My kernel parameters are:
gjjohnson@M93p:~/scylla-code-samples/mms$ sudo sysctl -p /etc/sysctl.conf
[sudo] password for gjjohnson:
fs.aio-max-nr = 2097152
fs.inotify.max_user_watches = 524288

I tried fs.aio-max-nr with the suggested value and was not able to start all three nodes, so I doubled the value to 2,097,152. There was no documentation on setting fs.inotify.max_user_watches, but in the setup document for for S110: The Mutant Monitoring System, the output from sysctl -p /etc/sysctl.conf shows:
ubuntu $ sysctl -p /etc/sysctl.conf
fs.inotify.max_user_watches = 524288
fs.aio-max-nr = 1048576
So, I set both parameters on my system.

I have 24GB of memory on this workstation and have set the Docker Desktop memory limit to 15GB.

On the node that fails to start, scylla-node3, the error message is:

024-01-24 00:25:27 FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)
2024-01-24 00:25:28 Traceback (most recent call last):
2024-01-24 00:25:28 File “/opt/scylladb/scripts/libexec/scylla-housekeeping”, line 196, in
2024-01-24 00:25:28 args.func(args)
2024-01-24 00:25:28 File “/opt/scylladb/scripts/libexec/scylla-housekeeping”, line 122, in check_version
2024-01-24 00:25:28 current_version = sanitize_version(get_api(‘/storage_service/scylla_release_version’))
2024-01-24 00:25:28 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 00:25:28 File “/opt/scylladb/scripts/libexec/scylla-housekeeping”, line 80, in get_api
2024-01-24 00:25:28 return get_json_from_url(“http://” + api_address + path)
2024-01-24 00:25:28 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-01-24 00:25:28 File “/opt/scylladb/scripts/libexec/scylla-housekeeping”, line 75, in get_json_from_url
2024-01-24 00:25:28 raise RuntimeError(f’Failed to get “{path}” due to the following error: {retval}')
2024-01-24 00:25:28 RuntimeError: Failed to get “http://localhost:10000/storage_service/scylla_release_version” due to the following error: <urlopen error [Errno 99] Cannot assign requested address>
2024-01-24 00:25:31 FATAL: Exception during startup, aborting: std::runtime_error (Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application)

I would really like to resolve this issue so I can proceed with the rest of this class.

Thanks!

GJ Johnson

roy · January 25, 2024, 11:26am

I’m trying to get a workaround so at least developer-mode won’t face this issue.

roy · January 25, 2024, 1:58pm

I’m happy to share that there is an easy workaround that works also on macOS:

You can start scylla with the following flag: “–reactor-backend=epoll”

For Example:

docker run --name some-scylla --hostname some-scylla -d scylladb/scylla --reactor-backend=epoll
docker run --name some-scylla2  --hostname some-scylla2 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"
docker run --name some-scylla3  --hostname some-scylla3 -d scylladb/scylla --reactor-backend=epoll --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' some-scylla)"

15:53 roy@macbook-pro ~ () $ docker exec -it some-scylla nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns    Host ID                               Rack
UN  172.17.0.3  369.39 KB  256          ?       dd207e02-468e-48c2-91ca-336cafff8f2f  rack1
UN  172.17.0.2  186.78 KB  256          ?       fa67ef85-468d-4c7c-8275-46bc3a6931e5  rack1
UN  172.17.0.4  566.11 KB  256          ?       444b5568-4500-410b-b1b0-2676c4f908cc  rack1

gjjohnson · January 25, 2024, 3:51pm

Thanks, Roy –

I was able to successfully start a 3-node cluster with your suggested change.

George J Johnson

gjjohnson · January 25, 2024, 10:25pm

So, for S110: The Mutant Monitoring System lab, the 3-node cluster is built using Docker Compose. Running the updated commands outside of Docker Compose will start the nodes successfully, but they will not be correctly configured for the labs.

I edited my local copy of ~/scylla-code-samples/mms/docker-compose.yml and after deleting the nodes on my cluster, I was able to rebuild them correctly using docker-compose.

The updated docker-compose.yml file is:

version: “3”

services:

scylla-node1:
container_name: scylla-node1
image: scylladb/scylla:5.2.0
restart: always
command: --seeds=scylla-node1,scylla-node2 --memory 750M --api-address 0.0.0.0 --reactor-backend=epoll
volumes:

“./scylla/scylla.yaml:/etc/scylla/scylla.yaml”
“./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties”
“./scylla/mutant-data.txt:/mutant-data.txt”
networks:
web:

scylla-node2:
container_name: scylla-node2
image: scylladb/scylla:5.2.0
restart: always
command: --seeds=scylla-node1,scylla-node2 --memory 750M --api-address 0.0.0.0 --reactor-backend=epoll
volumes:

“./scylla/scylla.yaml:/etc/scylla/scylla.yaml”
“./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties”
networks:
web:

scylla-node3:
container_name: scylla-node3
image: scylladb/scylla:5.2.0
restart: always
command: --seeds=scylla-node1,scylla-node2 --memory 750M --api-address 0.0.0.0 --reactor-backend=epoll
volumes:

“./scylla/scylla.yaml:/etc/scylla/scylla.yaml”
“./scylla/cassandra-rackdc.properties.dc1:/etc/scylla/cassandra-rackdc.properties”
networks:
web:

networks:
web:
driver: bridge

I don’t guarantee this is the best way to set up docker-compose.yml since I am not a Docker expert, but this does create a working 3-node cluster on my workstation. Thanks to everyone at Scylla for your help with this work-around! GJ Johnson