# Read errors in a cluster, IO setup failure, disk performance and optimizations **URL:** \ **Category:** ScyllaDB\ **Tags:** error-message, administration, configuration, io, disk\ **Created:** [August 4, 2025, 5:42am UTC](https://forum.scylladb.com/t/read-errors-in-a-cluster-io-setup-failure-disk-performance-and-optimizations/5033 "2025-08-04T05:42:14Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [August 4, 2025, 5:42am UTC](https://forum.scylladb.com/t/read-errors-in-a-cluster-io-setup-failure-disk-performance-and-optimizations/5033/1 "2025-08-04T05:42:14Z")
> [@](#): > > **[Originally from the User Slack](https://slack.scylladb.com/)** > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_WalErik-Jan\_van\_de\_Wal\*\*:\*\* Hi all, > I’m running into intermittent read errors under load on a 3-node Scylla 2025.1 cluster: > > Jul 31 16:01:02 learner01 taskset[121958]: Error read\_moving\_average\_data: Failed to fetch the first page of the result: Database returned an error: Not enough nodes responded to the read request in time to satisfy required consistency level (consistency: Quorum, received: 1, required: 2, data\_present: false) > > Cluster hardware: (servers are about 8 years old) > Node 1: 40 cores / 164 GB RAM / 20 TB disk (DELL PowerEdge R720) > Node 2: 20 cores / 164 GB RAM / 20 TB disk (DELL PowerEdge R720) > Node 3: 56 cores / 64 GB RAM / 20 TB disk (Supermicro) > > The servers are in the same rack, and the machine that runs the application that uses Scylla as well. > > Keyspace: > NetworkTopologyStrategy, RF = 3 > > Failing Table Schema: > > ```auto > CREATE TABLE ... ( > moving_average_num bigint, > moving_average double, > price double, > hour_identifier bigint, > timestamp bigint, > PRIMARY KEY ((moving_average_num, hour_identifier), timestamp) > ) WITH CLUSTERING ORDER BY (timestamp ASC) > AND compaction = { > 'class': 'TimeWindowCompactionStrategy', > 'compaction_window_unit': 'DAYS', > 'compaction_window_size': '1' > } > AND compression = { > 'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor' > }; > > ``` > > Query Pattern: > > ```auto > SELECT price, timestamp, moving_average, hour_identifier, moving_average_num > FROM keyspace.moving_average > WHERE moving_average_num = ? AND hour_identifier = ?; > > ``` > > Query function (Rust): > > ```auto > pub async fn get_moving_average_by_data_hour_identifier( > &self, > hour_identifier: i64, > moving_average_num: i64, > ) -> Result> { > let _permit = GLOBAL_QUERY_SEMAPHORE.acquire().await?; > > let prepared_values: (i64, i64) = (moving_average_num, hour_identifier); > > let mut rows_stream: TypedRowStream = self > .session > .execute_iter( > self.get_moving_average_by_data_hour_identifier.clone(), > prepared_values, > ) > .await? > .rows_stream::()?; > > let mut result: Vec = Vec::new(); > while let Some(next_row_res) = rows_stream.next().await { > result.push(next_row_res?); > } > > Ok(result) > } > > ``` > > **Load Profile:** > • App is a simulator running 8–10 jobs, each with max 15 sub-simulations in parallel > • Average: ~~7k ops/sec, peaks at~~ 10k ops/sec > • Errors occur mostly under high read pressure > Identified bottlenecks: CPU, disk > > Node 3 Diagnostics (Problem Node): > total\_successful\_reads: 1,469,616 > total\_failed\_reads: 3,234,429 > reads\_enqueued: 3,655,955 > reads\_queued\_count: 3,610,289 > permits: 100/100 > > Disk layout: > /dev/sda3 (LVM): > > - / (100G) > - /var/lib/scylla/commitlog (50G) > - /var/lib/scylla/hints (25G) > - /var/lib/scylla/view\_hints (10G) > - /var/lib/scylla/saved\_caches(25G) > > /dev/sdb1 (18.2TB): > > - /var/lib/scylla/data > > Note: `scylla_io_setup` failed on node 3 during install; I bybassed this by explicitly only test /var/lib/scylla/data and /var/lib/scylla/commitlog (because commitlog, hints, view\_hints, saved\_caches are the same disk) > > The problem mostly looks like node3, it looks like it can not keep up and node1 and node2 need to pick up the work for node3 where eventually node1 and node2 also crash. > > See the screenshot (the blue line is node3) > > Any help that push me in the right direction is helpful, I have exhausted everything I know so far. I do understand that node3 is the weakest link, and of course the first thing that needs to be solved is the scylla\_io\_setup, but I am stuck here > This is the error from scylla\_io\_setup > > coconut@scylla-03:~$ sudo scylla\_io\_setup > [sudo] password for coconut: > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:1/0:2:1:0/block/sdb/sdb1 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:1/0:2:1:0/block/sdb > already tuned: /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:1/0:2:1:0/block/sdb/queue/nomerges > tuning /sys/devices/virtual/block/dm-1 > tuning: /sys/devices/virtual/block/dm-1/queue/nomerges 2 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda3 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda > already tuned: /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda/queue/nomerges > tuning /sys/devices/virtual/block/dm-2 > tuning: /sys/devices/virtual/block/dm-2/queue/nomerges 2 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda3 > tuning /sys/devices/virtual/block/dm-4 > tuning: /sys/devices/virtual/block/dm-4/queue/nomerges 2 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda3 > tuning /sys/devices/virtual/block/dm-5 > tuning: /sys/devices/virtual/block/dm-5/queue/nomerges 2 > tuning /sys/devices/pci0000:80/0000:80:02.0/0000:81:00.0/host0/target0:2:0/0:2:0:0/block/sda/sda3 > INFO 2025-07-31 20:34:48,499 seastar - Reactor backend: io\_uring > INFO 2025-07-31 20:34:49,051 [shard 0:main] iotune - /var/lib/scylla/view\_hints passed sanity checks > INFO 2025-07-31 20:34:49,052 [shard 0:main] iotune - Disk parameters: max\_iodepth=916 disks\_per\_array=1 minimum\_io\_size=4096 > INFO 2025-07-31 20:34:49,056 [shard 0:main] iotune - Filesystem parameters: read alignment 512, write alignment 4096 > Starting Evaluation. This may take a while… > Measuring sequential write bandwidth: ERROR 2025-07-31 20:36:12,578 [shard 0:main] seastar - Exiting on unhandled exception: std::system\_error (error system:28, No space left on device) > ERROR:root:Command ‘[’/usr/bin/iotune’, ‘–format’, ‘envfile’, ‘–options-file’, ‘/etc/scylla.d/io.conf’, ‘–properties-file’, ‘/etc/scylla.d/io\_properties.yaml’, ‘–evaluation-directory’, ‘/var/lib/scylla/data’, ‘–evaluation-directory’, ‘/var/lib/scylla/commitlog’, ‘–evaluation-directory’, ‘/var/lib/scylla/hints’, ‘–evaluation-directory’, ‘/var/lib/scylla/view\_hints’, ‘–evaluation-directory’, ‘/var/lib/scylla/saved\_caches’, ‘–cpuset’, ‘1,2,3,4,5,6,7,8,9,10,11,12,13,15,16,17,18,19,20,21,22,23,24,25,26,27,29,30,31,32,33,34,35,36,37,38,39,40,41,43,44,45,46,47,48,49,50,51,52,53,54,55’]’ returned non-zero exit status 1. > ERROR:root:[‘/var/lib/scylla/data’, ‘/var/lib/scylla/commitlog’, ‘/var/lib/scylla/hints’, ‘/var/lib/scylla/view\_hints’, ‘/var/lib/scylla/saved\_caches’] did not pass validation tests, it may not be on XFS and/or has limited disk space. > This is a non-supported setup, and performance is expected to be very bad. > For better performance, placing your data on XFS-formatted directories is required. > To override this error, enable developer mode as follow: > sudo /opt/scylladb/scripts/scylla\_dev\_mode\_setup --developer-mode 1 > > It is only on the Scylla 3 node, it has the same harddrive capacity. I have reformatted the disks many times already > The partitions are formatted as XFS > > └─sda3 LVM2\_member LVM2 001 Ct4FCa-TwgT-4DjE-7dBR-6cgn-0b0B-MomIEC > ├─ubuntu–vg-ubuntu–lv ext4 1.0 60fe3131-3982-44ab-8698-b3895c152ca0 79.5G 14% / > ├─ubuntu–vg-scylla\_commitlog xfs addf9c73-d96a-43f3-80ab-cf7b40de31c2 20.8G 58% /var/lib/scylla/commitlog > ├─ubuntu–vg-scylla\_hints xfs 8249b669-f41e-4b2e-aabd-79fefe11a55f 24.4G 2% /var/lib/scylla/hints > ├─ubuntu–vg-scylla\_caches xfs 03a5ab30-3099-4b9a-b7b9-3bce7b6aca39 > ├─ubuntu–vg-scylla\_view\_hints xfs c8581434-4c36-4fcd-b287-ba9238f7e101 9.7G 2% /var/lib/scylla/view\_hints > └─ubuntu–vg-scylla\_saved\_caches xfs 23c34a60-8c32-4f10-9478-89616cc79a96 24.4G 2% /var/lib/scylla/saved\_caches > sdb > └─sdb1 xfs b00edbbb-08b8-4e7e-a77c-dfd81755f618 17.8T 2% /var/@Felipe\_Cardeneti\_Mendesib/scylla/data > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* Hm… there are quite a few problems here. > Node 3 has a smaller memory per vCPU than the rest of the nodes. Node1 ~~4GB/vCPU, Node2~~ 8GB/vCPU, Node 3 ~1GB/vCPU > Your `iotune` (what scylla\_io\_setup calls) fails due to: > > ```auto > INFO 2025-07-31 20:34:49,051 [shard 0:main] iotune - /var/lib/scylla/view_hints passed sanity checks > INFO 2025-07-31 20:34:49,052 [shard 0:main] iotune - Disk parameters: max_iodepth=916 disks_per_array=1 minimum_io_size=4096 > INFO 2025-07-31 20:34:49,056 [shard 0:main] iotune - Filesystem parameters: read alignment 512, write alignment 4096 > Starting Evaluation. This may take a while... > Measuring sequential write bandwidth: ERROR 2025-07-31 20:36:12,578 [shard 0:main] seastar - Exiting on unhandled exception: std::system_error (error system:28, No space left on device) > > ``` > > Given the sanity check is on `view_hints` I suppose that’s the directory it failed on. But why do you have a VG for each directory, rather than simply having a single `/var/lib/scylla` ? If these are slow disks, then maybe keep just the commitlog. > The 2% utilization under `/var/lib/scylla/data` also looks strange - is it all of your data, or is it simply not persisting any data? If the data is different than from the other nodes, then the read could be failing because it could be a large partiti@Erik-Jan\_van\_de\_Waln trying to read-repair to that node? > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* I though it would be wise to keep my actual data on a seperate disk (the 20TB) and keep the other data on@Felipe\_Cardeneti\_Mendesthe primary (OS) disk. As recommended in the config file > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* if at all possible,@Erik-Jan\_van\_de\_WalI’d suggest you try to keep similar shard count and memory ratios per vCPU. Hmm > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* the 2% utlization is probably correct. I only have about 5 millions records > I need to work with the hardware I have (wooh@Felipe\_Cardeneti\_Mendeso startups). Would it be better if I virtualize (docker) Scylla per server to make them identical? > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* \> I though it would be wise to keep my actual data on a seperate disk (the 20TB) and keep the other data on the primary (OS) disk. As recommended in the config file > Yeah, if these disks are NVMes you should be good with just a single mount. The separate filesystem recommendation for the commitlog is mai@Erik-Jan\_van\_de\_Wally when the disks are known to be slow, which is no@Felipe\_Cardeneti\_Mendes the case with NVMes (hints are rarely used, and we never use **caches** ) > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* Right. The disks are magnetic disks. Not SSDs > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* \> Would it be better if I virtualize (docker) Scylla per server to make them identical? > Or just reduce the shard count (`/etc/scylla.d/cpuset.conf`). You can also reduce memory in `/etc/scylla.d/memory.conf`. > > We often use ~8GB per vCPU. 4GB works, but I wouldn’t go lower than that. The more memor@Erik-Jan\_van\_de\_Wal, the more caching space you have, plus headroom for metadata. > oh ok, with magnetic, definitely keep the commitlog, b@Felipe\_Cardeneti\_Mendest drop the rest, you don’t need these LVs. 🙂 > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* got it! > so I take the weakest link, and make sure that the other servers are locked to that config too 8GB/vCPU > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* also, increase the commitlog LV size to about the memory size you have, otherwise recycling their segments may be too frequent. > yeah, 8 vCPUs should work fine for that amount of RAM > one last @Erik-Jan\_van\_de\_Walote before you move forward - since you’re on 2025.1 - **IF** you are using tablets, then you’d @Felipe\_Cardeneti\_Mendesee@avi to replace the node. Or j@Erik-Jan\_van\_de\_Walst recreate the clu@avit@Felipe\_Cardeneti\_Mendesr with the right configs if you are ok with losing the data. > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* Thanks@Felipe\_Cardeneti\_Mendes I opted out for tablets so I should be good with reconfiguring them one by one > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* @avi drops a tear silently > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* aaaaaw i’m sorry @avi > @Felipe\_Cardeneti\_Mendes could I@Erik-Jan\_van\_de\_Walalso pass @Felipe\_Cardeneti\_Mendeshe cpu and mem config via SCYLLA\_ARGS in /etc/default/scylla-server? > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* yes > all these are sourced by the systemd scylla-server unit, so ensure you dont have duplicates so the cmdline isn’t messed up. But other than that, you have the flexibility to adjust it as you see better fit. > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* Thanks! > @Felipe\_Cardeneti\_Mendes so I did what you suggested. > > • Removed the LVMS > • Increased the commit partition > • Configured each server to use 8CPU/64GB RAM > And I still get the following error on only one (the 3rd) server. > > ```auto > ug 01 03:04:09 scylla-03 scylla[107456]: [shard 0:sl:d] reader_concurrency_semaphore - (rate limiting dropped 736 similar messages) Semaphore sl:default with 100/100 count and 2861400/155398963 memory resources: timed out, dumping permit diagnostics: > Trigger permit: count=1, memory=23979, table=data.price_0_0002, operation=data-query, state=active/await > Identified bottleneck(s): CPU, disk > > permits count memory table/operation/state > 34 34 1168K data.regression/data-query/active/await > 42 42 1071K data.moving_average/data-query/active/await > 24 24 556K data.price_0_0002/data-query/active/await > 24 0 0B data.price_0_0002/data-query/waiting_for_admission > 117 0 0B data.moving_average/data-query/waiting_for_admission > 46 0 0B data.regression/data-query/waiting_for_admission > > 287 100 2794K total > > Stats: > permit_based_evictions: 0 > time_based_evictions: 0 > inactive_reads: 0 > total_successful_reads: 272 > total_failed_reads: 13848 > total_reads_shed_due_to_overload: 0 > total_reads_killed_due_to_kill_limit: 0 > reads_admitted: 3409 > reads_enqueued_for_admission: 14397 > reads_enqueued_for_memory: 0 > reads_admitted_immediately: 10 > reads_queued_because_ready_list: 3 > reads_queued_because_need_cpu_permits: 1296 > reads_queued_because_memory_resources: 0 > reads_queued_because_count_resources: 13098 > reads_queued_with_eviction: 0 > total_permits: 14407 > current_permits: 287 > need_cpu_permits: 100 > awaits_permits: 100 > disk_reads: 100 > @Felipe_Cardeneti_Mendes sstables_read: 138 > > ``` > > As you can s@Erik-Jan\_van\_de\_Wale in the screenshot blue is completly flatlined. The problem so far is that when I have a low req/s everything seems to go well, but when I got to 5K+ req/s stuff starts to break. > > Any other advice? I tried to lower the num\_tokens to 128, but that doesnt change anything. > The load is also low > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* what’s the `io_properties.yaml` values compared to other nodes? > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* server\_1: > > coconut@scylla-01:/etc/scylla.d$ cat io\_properties.yaml > disks: > > - mountpoint: /var/lib/scylla/commitlog > read\_iops: 232 > read\_bandwidth: 84852776 > write\_iops: 1198 > write\_bandwidth: 209791600 > - mountpoint: /var/lib/scylla/data > read\_iops: 231 > read\_bandwidth: 191295152 > write\_iops: 1305 > write\_bandwidth: 254318288 > > server\_2: > > coconut@scylla-02:/etc/scylla.d$ cat io\_properties.yaml > disks: > > - mountpoint: /var/lib/scylla/commitlog > read\_iops: 248 > read\_bandwidth: 87394640 > write\_iops: 1222 > write\_bandwidth: 210555792 > - mountpoint: /var/lib/scylla/data > read\_iops: 234 > read\_ban@Felipe\_Cardeneti\_Mendeswidth: 92834344 > write\_iops: 1275 > write\_bandwidth: 252885664 > > server\_3 (problem server): > > coconut@scylla-03:/etc/scylla.d$ cat io\_properties.yaml > disks: > > - mountpoint: /var/lib/scylla/commitlog > read\_iops: 245 > read\_bandwidth: 162483360 > write\_iops: 3557 > write\_bandwidth: 213447312 > - mountpoint: /var/lib/scylla/data > read\_iops: 129 > read\_bandwidth: 174533520 > write\_iops: 4058 > write\_bandwidth: 226609440 > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) @Felipe\_Cardeneti\_Mendes\*\*:\*\* Yeah, it’s definitely disk. The semaphore dump earlier shows all read permits are consumed (100/100) with reads in progress. Once the queue gets full, other requests accumulate until they timeout or we kill it due to overload. > > Is it possible most reads from n1/n2 are served from cache? You can check the misses/hits in the Cache section within the Detailed panel. > It is probably wise to see the IOPS metrics on the Advanced dashboard for your classes (memtable, compaction, sl:default) and compare these nodes. > > My wild guess is that your reads on N3 are saturating its IOPS, and the course of action (despite using faster disks) would be to tune the system to use a less IOPS as possible. This would involve `chunk_length_kb` ([https://www.scylladb.com/2017/08/01/compression-chunk-sizes-scylla/](https://www.scylladb.com/2017/08/01/compression-chunk-sizes-scylla/)) depending on your average read size, bloom filters, and (at the extreme) SSTable summary ratios ([https://medium.com/agoda-engineering/exploring-scylla-disk-performance-and-optimizations-at-agoda-65a6dcdd6fe7](https://medium.com/agoda-engineering/exploring-scylla-disk-performance-and-optimizations-at-agoda-65a6dcdd6fe7)). > > You can also play with different Seastar settings - like io-latency-goal-ms ([https://www.scylladb.com/2023/07/17/top-mistakes-with-scylladb-storage/](https://www.scylladb.com/2023/07/17/top-mistakes-with-scylladb-storage/)): > > ```auto > Defines how long requests to disk should take. Defaults to 1.5 * task-quota-ms. The value should be great@Erik-Jan_van_de_Walr than a single request's latency, as it allows for more requests to be dispatched simultaneously. For spinning disks with an average latency of 10ms, increasing the latency goal to at least 50ms should allow for some concurrency. > > ``` > > Lastly, triple check the number of CPU/shards stand as you configured them - `/usr/lib/scylla/seastar-cpu-map.sh -n scylla` > > > Medium: Exploring Scylla Disk Performance and Optimizations at Agoda > > ![Erik-Jan_van_de_Wal](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/4/439e289ccc7434e52870b6a6f538b223165d7f00.jpeg) @Erik-Jan\_van\_de\_Wal\*\*:\*\* Thanks. I will research further and dive deeper in the documentation. I think this is a case of “never let a software engineer handle database engineering”. > > We can run our tests and potentially move to live. I’ve built a L1 cache between the application and data and it seems to hold (I also removed N3 from the cluster and put RF=2, not ideal, but for now we need to work with what we’ve got) > > Thanks again for the quick response and helpful insights @Felipe\_Cardeneti\_Mendes!