Originally from the User Slack
@hamonicahamonica**:** I am planning to store observability data using ScyllaDB 6.2, but I am experiencing lower-than-expected performance and would like to request technical guidance.
Under normal conditions, writing 10,000–20,000 rows per second is not a problem. These rows represent what I call root spans.
However, each root span has child spans beneath it—anywhere from fewer than 10 to as many as 100–300 child spans. Assuming an average of 150 child spans per root span, I am observing significant disk I/O wait during writes.
I created a span_trace table as shown below. The data is primarily grouped by trace_id. However, if I include trace_id as part of the partition key, I understand that this can lead to an excessive number of partitions, which may also be problematic.
I am trying to understand how to improve write performance under this model.
I have experimented with bucketing created_at at various granularities—5 seconds, 10 seconds, and currently 1 minute and 5 minute buckets—but I am still testing to find the optimal configuration.
To clarify, the 10k–20k rows/sec figure represents an upper-bound scenario. At present, the actual workload is closer to 1,000–2,000 TPS (rows per second).
CREATE TABLE IF NOT EXISTS kstal.span_trace (
workspace_no bigint,
created_at bigint,
service_name text,
trace_id text,
service_instance_id text,
span_id text,
parent_span_id text,
name text,
start_time_unix_nano bigint,
end_time_unix_nano bigint,
status text,
kind text,
attributes frozen<list<frozen<attribute>>>,
events text,
PRIMARY KEY ((workspace_no, created_at, service_name), trace_id, span_id)
)
WITH DEFAULT_TIME_TO_LIVE = 1209600 -- 2 weeks
AND compaction = {
'class': 'TimeWindowCompactionStrategy',
'compaction_window_unit': 'DAYS',
'compaction_window_size': 1
};
OS : Rocky-Linux-9
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 52 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Vendor ID: AuthenticAMD
BIOS Vendor ID: QEMU
Model name: AMD EPYC 9224 24-Core Processor
mem: @dor6gb
@dor**:** Well, you run in a VM. It’s ok but you need to make sure that the VM cpus/memory aren’t taken by other apps on the host. Ideally pin them. In addition, you must run scylla setup at the beginning