Installation details
#ScyllaDB version: 2025.1.2-0.20250422.502c62d91d48
#Cluster size: 128 core
os (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu 22.04.5 LTS
Scylla doesn’t start because the iotune checks are failing. Deeper inspection showed that they were running the check assuming all Cpu cores on the node was available for it. I have too many cores on the node and can’t dedicate all of them to scylla. I exec-ed into the scylla container and manually ran the iotune test but with the smp flag /usr/bin/iotune --format envfile --options-file /etc/scylla.d/io.conf --properties-file /etc/scylla.d/io_properties.yaml --evaluation-directory /var/lib/scylla/data --default-log-level debug --smp=1
This causes the scylla pod to start working. I have to repeat the above process for all pods in my statefulset.
I have been able to replicate this many times in different clusters.
E0624 09:37:05.566229 1 sidecar/controller.go:156] "Unhandled Error" err="syncing key 'scylla/scylla-local-r1-1' failed: can't sync the HostID annotation: can't get HostID: can't get local HostID: Get \"http://localhost/storage_service/hostid/local\": dial tcp [::1]:10000: connect: connection refused" logger="UnhandledError"
ERROR:root:Command '['/usr/bin/iotune', '--format', 'envfile', '--options-file', '/etc/scylla.d/io.conf', '--properties-file', '/etc/scylla.d/io_properties.yaml', '--evaluation-directory', '/var/lib/scylla/data', '--**cpuset', '0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127']**' died with <Signals.SIGABRT: 6>.
ERROR:root:['/var/lib/scylla/data'] did not pass validation tests, it may not be on XFS and/or has limited disk space.
This is a non-supported setup, and performance is expected to be very bad.
For better performance, placing your data on XFS-formatted directories is required.
To override this error, enable developer mode as follow:
sudo /opt/scylladb/scripts/scylla_dev_mode_setup --developer-mode 1
tuning /sys/devices/rbd/14/block/rbd14
tuning: /sys/devices/rbd/14/block/rbd14/queue/nomerges 2
warning: unable to tune /sys/devices/rbd/14/block/rbd14/queue/nomerges to 2
failed!
Traceback (most recent call last):
File "/docker-entrypoint.py", line 27, in <module>
setup.io()
File "/scyllasetup.py", line 68, in io
self._run(['/opt/scylladb/scripts/scylla_io_setup'])
File "/scyllasetup.py", line 38, in _run
subprocess.check_call(*args, **kwargs)
File "/opt/scylladb/python3/lib64/python3.12/subprocess.py", line 415, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/opt/scylladb/scripts/scylla_io_setup']' returned non-zero exit status 1.```