Hello.
I’m using ScyllaDB Open Source 6.2.3 on physical machines.
I’m running scylla_setup command :
scylla_setup --no-raid-setup --no-selinux-setup --no-ntp-setup --no-coredump-setup --no-sysconfig-setup --io-setup 1 --no-version-check --no-fstrim-setup --no-rsyslog-setup
I have NVME disks on my ScyllaDB servers organized like this :
nvme0n1 259:0 0 1.7T 0 disk
├─data_commit_vg-6la_commitlog_lv 253:13 0 1000G 0 lvm /var/lib/scylla/commitlog
├─data_commit_vg-6la_hints_lv 253:16 0 250G 0 lvm /var/lib/scylla/hints
└─data_commit_vg-6la_view_hints_lv 253:18 0 100G 0 lvm /var/lib/scylla/view_hint
nvme2n1 259:1 0 1.7T 0 disk
└─data_vg-6la_data_lv 253:15 0 1.7T 0 lvm /var/lib/scylla/data
Unfortunately, the scylla_setup command fails because iotune saturates the disks it tests. Here is the iotune command performed :
/usr/bin/iotune --format envfile --options-file /etc/scylla.d/io.conf --properties-file /etc/scylla.d/io_properties.yaml --evaluation-directory /var/lib/scylla/data --evaluation-directory /var/lib/scylla/commitlog --evaluation-directory /var/lib/scylla/hints --evaluation-directory /var/lib/scylla/view_hints
Here is the error:
/opt/scylladb/scripts/libexec/scylla_io_setup:41: SyntaxWarning: invalid escape sequence ‘\s’
pattern = re.compile(_nocomment + r"CPUSET=\s*"" + _reopt(_cpuset) + _reopt(_smp) + “\s*"”) ` WARN 2025-04-24 10:54:19,580 seastar - Kernel memory reservation (/proc/sys/vm/min_free_kbytes) unexpectedly high (5497558016), check your configuration
INFO 2025-04-24 10:54:19,674 seastar - Reactor backend: linux-aio
INFO 2025-04-24 10:54:20,415 [shard 0:main] iotune - /var/lib/scylla/view_hints passed sanity checks
INFO 2025-04-24 10:54:20,416 [shard 0:main] iotune - Disk parameters: max_iodepth=1023 disks_per_array=1 minimum_io_size=4096
INFO 2025-04-24 10:54:20,420 [shard 0:main] iotune - Filesystem parameters: read alignment 512, write alignment 4096
ERROR 2025-04-24 10:54:35,453 [shard 0:main] seastar - Exiting on unhandled exception: std::system_error (error system:28, No space left on device)
ERROR:root:Command ‘[’/usr/bin/iotune’, ‘–format’, ‘envfile’, ‘–options-file’, ‘/etc/scylla.d/io.conf’, ‘–properties-file’, ‘/etc/scylla.d/io_properties.yaml’, ‘–evaluation-directory’, ‘/var/lib/scylla/data’, ‘–evaluation-directory’, ‘/var/lib/scylla/commitlog’, ‘–evaluation-directory’, ‘/var/lib/scylla/hints’, ‘–evaluation-directory’, ‘/var/lib/scylla/view_hints’]’ returned non-zero exit status 1.
ERROR:root:[‘/var/lib/scylla/data’, ‘/var/lib/scylla/commitlog’, ‘/var/lib/scylla/hints’, ‘/var/lib/scylla/view_hints’] did not pass validation tests, it may not be on XFS and/or has limited disk space.
This is a non-supported setup, and performance is expected to be very bad.
For better performance, placing your data on XFS-formatted directories is required.
To override this error, enable developer mode as follow:
sudo /opt/scylladb/scripts/scylla_dev_mode_setup --developer-mode 1
Disks are correctly formatted in XFS.
I tried to decrease duration because by default it is 120s, but even with 20 seconds, iotune saturates view_hints disk and fails.
Why does iotune saturate the disk pls ? Shoudn’t it stop its tests when it arrives at 80-90% of the disk ? What can I do to solve this problem and be able to execute scylla_setup ?
Best regards.