Need some help in troubleshooting token-aware requests. I am using 5.2.2 ScyllaDB with 3.x java Scylla driver. The requests are reaching the right node, but the node does not read the local replica and instead requests another host. I have confirmed this with the tracing enabled, the target selected is not the local node.
51d635f0-3b8b-11ee-b96e-44d4dd97d627 | 10.220.4.28 | shard 6 | Creating read executor for token 8807409880602549597 with all: {10.220.4.28, 10.220.8.254, 10.220.7.248} targets: {10.220.7.248} repair decision: NONE
Is there anything needed to avoid an additional hop?
Is this a paged read? Paged reads will stick to the same set of replicas for the duration of the query, to be able to re-use saved readers on said readers. Doing so has more benefits than avoiding the hop.
Thanks for the reply. This is not a paged read. It is a lookup for a specific partition and reads all rows in the partition.
I checked the code and another thing that can force a non-local replica is cache hit-rate based load balancing. This tries to send more requests to nodes that already have warmed-up caches, slowly ramping up newly started nodes. Did you restart any of the nodes recently? Can you still observe the hop after some time has passed?
The nodes have been up and running for a while. The additional hop is observed consistently. To rule out cache hit-rate-based load balancing, is there a configuration for disabling that feature?
What does your query look like?
Yes, it is called cache_hit_rate_read_balancing.
Thanks will give it a try by disabling cache_hit_rate_read_balancing. For the query, I am retrieving the entire partition select * from tableX where partition_key = 'x'