ScyllaDB timeout error comes while writing/reading from/to table

ScyllaDB
1

We are using ScyallDB version 3.0.8 in a production environment with 3 nodes cluster. Each node has 32-core CPU 128GB RAM.
ScyllaDB has 1 DB with an hourwise partition table. Each hour table max contains 30M rows, with each row having 4 columns containing email information ( TxId, from_address,to_address, maile_content). Each Row has max 50KB size.
We have been facing a timed-out error in some time while reading/writing to the disk. I have configured the timed out error. in scyall.yml is 10 seconds.
Below is my nodetool proxyhistograms output
Percentile Read Latency Write Latency Range Latency CAS Read Latency CAS Write Latency View Write Latency
(micros) (micros) (micros) (micros) (micros) (micros)
50% 378.00 542.50 2546.00 0.00 0.00 0.00
75% 496.00 169026.50 3107.00 0.00 0.00 0.00
95% 943.75 464572.00 4862.75 0.00 0.00 0.00
98% 1112.50 549202.00 10827.00 0.00 0.00 0.00
99% 2569.75 1820080.00 31794.50 0.00 0.00 0.00
Min 64.00 66.00 1604.00 0.00 0.00 0.00
Max 12925.00 10000085.00 2013221.00 0.00 0.00 0.00

why I am getting timedout error, I am looking for read and write latency <=1 milli seconds. How can I achiev this?

2

Why are you using such an old ScyllaDB version? Please try with 5.4 (or the latest)

3

First, as Guy wrote above, this is a very very old scylladb version.
Latest OSS release is 5.4.1.
I suggest you to use that (though, your upgrade path isn’t going to be simple).
(Maybe a backup and restore would be easier, but still not sure clean it would be).

Second, how does your reads look like?

Third, IIUC, you have partitions with 30M rows, with 50K each row?
Does it mean you have 1.43 TB size of a partition??
In that case, your data model is not designed well and your target latency impossible.
You must redesigning your data model (based on how would want to read the data).

Forth, how big is user dataset? how much data do you save?
I guess you’re not using TWCS, I’m not even sure it was introduced in this release.

1 Like