Iotune tests fails because it saturates evaluation disks

Hello.

I’m using ScyllaDB Open Source 6.2.3 on physical machines.

I’m running scylla_setup command :

scylla_setup --no-raid-setup --no-selinux-setup --no-ntp-setup --no-coredump-setup --no-sysconfig-setup --io-setup 1 --no-version-check --no-fstrim-setup --no-rsyslog-setup

I have NVME disks on my ScyllaDB servers organized like this :

nvme0n1 259:0 0 1.7T 0 disk
├─data_commit_vg-6la_commitlog_lv 253:13 0 1000G 0 lvm /var/lib/scylla/commitlog
├─data_commit_vg-6la_hints_lv 253:16 0 250G 0 lvm /var/lib/scylla/hints
└─data_commit_vg-6la_view_hints_lv 253:18 0 100G 0 lvm /var/lib/scylla/view_hint
nvme2n1 259:1 0 1.7T 0 disk
└─data_vg-6la_data_lv 253:15 0 1.7T 0 lvm /var/lib/scylla/data

Unfortunately, the scylla_setup command fails because iotune saturates the disks it tests. Here is the iotune command performed :

/usr/bin/iotune --format envfile --options-file /etc/scylla.d/io.conf --properties-file /etc/scylla.d/io_properties.yaml --evaluation-directory /var/lib/scylla/data --evaluation-directory /var/lib/scylla/commitlog --evaluation-directory /var/lib/scylla/hints --evaluation-directory /var/lib/scylla/view_hints

Here is the error:

/opt/scylladb/scripts/libexec/scylla_io_setup:41: SyntaxWarning: invalid escape sequence ‘\s’
pattern = re.compile(_nocomment + r"CPUSET=\s*"" + _reopt(_cpuset) + _reopt(_smp) + “\s*"”) ` WARN 2025-04-24 10:54:19,580 seastar - Kernel memory reservation (/proc/sys/vm/min_free_kbytes) unexpectedly high (5497558016), check your configuration
INFO 2025-04-24 10:54:19,674 seastar - Reactor backend: linux-aio
INFO 2025-04-24 10:54:20,415 [shard 0:main] iotune - /var/lib/scylla/view_hints passed sanity checks
INFO 2025-04-24 10:54:20,416 [shard 0:main] iotune - Disk parameters: max_iodepth=1023 disks_per_array=1 minimum_io_size=4096
INFO 2025-04-24 10:54:20,420 [shard 0:main] iotune - Filesystem parameters: read alignment 512, write alignment 4096
ERROR 2025-04-24 10:54:35,453 [shard 0:main] seastar - Exiting on unhandled exception: std::system_error (error system:28, No space left on device)
ERROR:root:Command ‘[’/usr/bin/iotune’, ‘–format’, ‘envfile’, ‘–options-file’, ‘/etc/scylla.d/io.conf’, ‘–properties-file’, ‘/etc/scylla.d/io_properties.yaml’, ‘–evaluation-directory’, ‘/var/lib/scylla/data’, ‘–evaluation-directory’, ‘/var/lib/scylla/commitlog’, ‘–evaluation-directory’, ‘/var/lib/scylla/hints’, ‘–evaluation-directory’, ‘/var/lib/scylla/view_hints’]’ returned non-zero exit status 1.
ERROR:root:[‘/var/lib/scylla/data’, ‘/var/lib/scylla/commitlog’, ‘/var/lib/scylla/hints’, ‘/var/lib/scylla/view_hints’] did not pass validation tests, it may not be on XFS and/or has limited disk space.
This is a non-supported setup, and performance is expected to be very bad.
For better performance, placing your data on XFS-formatted directories is required.
To override this error, enable developer mode as follow:
sudo /opt/scylladb/scripts/scylla_dev_mode_setup --developer-mode 1

Disks are correctly formatted in XFS.

I tried to decrease duration because by default it is 120s, but even with 20 seconds, iotune saturates view_hints disk and fails.

Why does iotune saturate the disk pls ? Shoudn’t it stop its tests when it arrives at 80-90% of the disk ? What can I do to solve this problem and be able to execute scylla_setup ?

Best regards.

Hi Gwenael, welcome to the ScyllaDB Community Forum!

We don’t recommend separating ScyllaDB data components into separate partitions. You’ll likely end up wasting IO bandwidth by doing such splits as most of them will be idle most of the time.

Our recommended setup is to RAID0 all available disks (assuming they have the same capacity and performance characteristics) into a single partition and mount it as /var/lib/scylla.
This can be done via the scylla_setup process, or separately via the scylla_create_raid script.

This way, all subdirectories are created within that device and ScyllaDB can leverage the RAID IO bandwidth to its maximum.

You can explore more on our disks recommendations on this blog.

1 Like

Hello Gui.

Yes it is like this.
The data is on multiple disks.
But there is one disk used for commitlog, hints and view hints.

And the problem of disk saturation occures on view hints because I set the FS to 100 GB.