Hi Scylla Team,
So come back after month and my project is up again.
TLDR; I wanted to replace my C* cluster by Scylla (hoping for perf gain and cost reduction).
Context:
I have 3 clusters (different sizing depending on region).
The dataset can be big :
the most problematic ks/table is 3 billions rows with 1.3 new billions row a day (and then if you do the math quick expiration with TTL)
On the biggest region my setup is currently (C* 3.1) :
15 x c6id.8xlarge (32 cores 64g ram local nvme)
Peak read/s : 700K requests / s
Write : nothing expect when loading (done by sstable loader mostly take 2H)
On smalests regions :
8 x c7a.4xlarge (16 cores 32g ram EBS ; I know)
Peak read/s : 50K requests / s
Write : nothing expect when loading (done by sstable loader mostly take 2H)
I did test. A lot.
Using scylla 6.1.2.
So my biggest problem is how loading my 1.2 billions row dataset in a reasonable amount of time. On C* sstableloader stream my already made sstables and stream them fast enough to the right shard.
My scylla test cluster is currently :
4x i4i.4xlarge (16cpu, 64g ram, NVME) an instance which is advised by the doc.
On scylla sstableloader make sstable to cql translation which is slow as hell.
I know this is adviced to use a lot of thread and //ize the tool but even I’ve got bad result (20k insert per second, do the math it will take age)
What I tried also as insertion method :
- a custom golang sstable loader (which is mostly doing the samething as sstableloader from scylla) but I wanted to give a shoot by myself and see. I had better result with 48586k insert/s. Playing with number of // workers and in flight request (I can provide the code if you want). I see something very strange I want to understand. At some point scylla seems to not accept insert anymore (or maybe it’s problem client side) ; and then start again. Sometime it’s even trigger failling the insert (50 retries :/) . No log on scylla server side.
I really wonder if I missed something : like an obvious settings ? or a way to debug
Also I tried nodetool refresh load-and-stream and results are also a bit dissapointing :
load_and_stream: finished ops_uuid=1f41fcd5-057d-4f64-9917-232df78d2272, ks=ks_user_counters, table=user_counters_batch, partitions_processed=1164535 partitions, bytes_processed=5385636132 bytes, partitions_per_second=29578.027 partitions/s, bytes_per_second=130.45291 MiB/s, duration=39.371624 s, status=succeeded
so again maybe I miss something obvious ?
by the way do you know if there is tooling arround nodetool refresh?
Where I was surprised is why I am so far from theoretical qps stated on the site.
12.5K per core ; so I my setup 32 cores (64 vcpus) RF=1 I should be around 400k so I am at 15% expectations at best
Again I have the feeling that I make something very bad.
Maybe something related to the schema of the table (81 columms)? I read this something bad. Does this hit the performance that bad?
Anyway I take any advices; without help I will stuck with C* 