I would love any feedback on this blog that I just posted. Is the general idea correct? Am I missing anything important? Am I actually misleading anyone? Thanks
Ahoj Rado
vyzera to dobre 
I would love you to explain “magic numbers” … (you start, not sure if it’s fully clear)
We generally say here to have max 10-30 foreground requests per cpu, total concurrency is just a function of number of cpus per node and number of nodes multiplied to how big can be the background queue of queries to still fit your SLAs
Otherwise great explanation how spark works (or should work, since it doesn’t go down to shards fully, I have a repo for shard aware spark connector, but didn’t fully test if it will get close to logic you drawn)
Numberly also came close with How Numberly Replaced Kafka with a Rust-Based ScyllaDB Shard-Aware Application - ScyllaDB (and I saw few go and python ideal full scan implementation too, but without focus on per shard throttling).
Shards also split and optimize access to disk using schedulers (we tried to explain them in blogs).
Also routing token if you have metadata should be set dynamically by driver, why do you explicitely set, am I missing something? ( DataStax Java Driver - Load balancing )
Also spark uses shuffle to randomly shuffle tasks with ranges so they don’t get blocked on same shard, but I see you are trying to improve it - and btw. this is crucial - finding balance between too many small tasks and bigger tasks they try not to overlap is key, even for spark.
This partiitoning to tasks/token ranges is something which can be optimized everywhere, spark gives nice task distribution engine, but this partiitoning is something that can be tuned in similar way as you try, so keep it up!
Looking forward to next version! (or taking these ideas to spark?)
xing the fingers
L
1 Like