Originally from the User Slack
@Igor_Q: Hi again, everyone.
I have a cluster of 3 DC x 3 nodes of ScyllaDB 6.0.4. Table has the following schema:
CREATE TABLE items (
id bigint,
field text,
data blob,
updated_at timestamp,
PRIMARY KEY (id, field)
)
Keyspace has a replication factor of 1. At the moment we’re using it to read data by ID. Each request contains 1500 IDs, we split them into batches of 75 IDs and request them from Scylla in parallel (WHERE id IN ?). We’re using the fork of gocql.
Now, I’ve read in multiple places, that it’s generally advisable to request data by a single key in parallel (WHERE id = ?) – this would allow the driver to select the correct shard by token-aware policy and remove the coordination load from Scylla. I’m exploring the possibility of moving to such implementation, but I see weird results in the benchmark.
BenchmarkSelect_Batch_20-80 2648 12912375 ns/op // select 1500 keys in batches of 75 in parallel
BenchmarkSelect_ByOne_1500-80 1440 25113931 ns/op // select 1500 keys by one in 1500 goroutines
BenchmarkSelect_ByOne_75_x2-80 1459 23062871 ns/op // select 1500 keys by one in parallel in two sessions with 75 goroutines in each
First of all, selecting the keys by one does not come near reading them in batches. Query traces nevertheless show that the queries are executed by the owning replica. Is network solely responsible for such difference in latency?
Second, selecting keys by one in 1500 goroutines shows consistently slower results than creating two sessions and using only 75 goroutines in each (!). Is there some contention in *gocql.Session?
What am I doing wrong?
GitHub: GitHub - scylladb/gocql: Package gocql implements a fast and robust ScyllaDB client for the Go programming language.
@avi: You have one node per DC, so the batches always land on the right node. You save same CPU by using a batch, and don’t lose anything due to queries that have to be rerouted, so batches win.
In normal clusters that have more nodes, sending individual queries is better.
About gocql, I don’t know enough to comment.
@Igor_Q: > You have one node per DC
3 DC x 3 nodes = 3 nodes in each datacenter
Total of 9 nodes across all datacenters.
@avi: Then I don’t have an explanation, maybe it’s gocql.