I want to make clear my understanding about scylla java driver 4.x.
= Background =
We have a service application running in k8s with auto scale deployment in AWS VPC1, which uses Cassandra java driver 3.x to access Scylla enterprise cluster which runs in AWS VPC2. So actually there is a k8s NAT reside between the two systems.
Because Scylla java driver can provide Shard-Aware feature comparing with Cassandra driver, so we decide to migrate application to use Scylla java driver.
Now we have chosen Scylla java driver 4.15.0.1, and tested the migrated application to connect Scylla enterprise cluster port 19042, all looks good.
= question =
We are not sure whether the Shard-Aware feature works or not in our case. From the driver’s public doc, I cannot find answers. And there are no useful posts on internet to clarify.
Why we have such a question?
1, ScyllaDB can provide normal service even Shard-Aware feature not work.
2, Scylla java driver 4.x doc does not state it is shard aware.
3, No public doc clarifies the relationship between Shard-Aware feature and the different serving port (9042 vs 19042). We are confused if we should use 19042 or 9042.
4, There is a public post (Connect Faster to ScyllaDB with a Shard-Aware Port - ScyllaDB) saying that Shard-Aware feature does not work when the application is behind a NAT, but from Scylla java driver 4.x code, the logic looks like the Shard-Aware feature can work even there is a NAT.
So we are totally confused right now.
In other words, we want to know which statements are true:
1, Scylla java driver 4.x is not Shard-Aware
2, Scylla java driver 4.x is Shard-Aware, but the feature does not work behind NAT
3, Scylla java driver 4.x is Shard-Aware and the feature works behind NAT
4, Scylla java driver 4.x must use 19042 to make Shard-Aware feature work
5, Scylla java driver 4.x can use 9042 or 19042, and both ports work for Shard-Aware feature.
Shard Awareness: the ability of a driver to route a request to a shard that owns the data. It requires the driver to have a connection open to the shard. This is why driver tries to open a connection to each shard of each node. When you connect to normal port (9042) you get assigned “random” shard (I think it’s not really random, but that’s not important here) - so opening connection to each one can take a lot of failed attempts and a lot of time.
Advanced Shard Awareness - the ability to open a connection to a specific shard, so that building a full connection pool is faster. This is where port 19042 is used. When you make a connection to it, Scylla looks at the source port of this connection and assigns it to a shard number source_port % amount_of_shards. That allows the driver to not have any failed attempts when building connection pool.
When there is NAT between Scylla and the driver, source port visible by server will be different than the one that the client really uses - so Advanced Shard Awareness won’t work. Drivers that support Advanced Shard Awareness will detect this situation, fall back to port 9042 and build connection port the old way. Shard Awareness will still work in this situation, but fully creating a session will just take longer.
If I remember correctly, Java Driver 4.x support Shard Awareness but not Advanced Shard Awareness - @piotr or @Bouncheck please correct me if I’m wrong.
So to answer your questions:
1, Scylla Java Driver 4.x is Shard Aware, but does not have Advanced Shard Awareness (for now - because we plan to implement it).
2, Shard Awareness works behing NAT. Advanced Shard Awareness does not, but 4.x does not have it for now anyway.
3, Yes
4, No, Shard Awareness does not depend on the port. Only Advanced Shard Awareness does.
5, Correct.
Thank you Lorak, you saved me. Your clarification is what I want.
Without your explanation, I don’t know there are two terms for shard-aware, because there are no public docs mentioning it.
Thank you very much.
When there is NAT between Scylla and the driver … Shard Awareness will still work in this situation, but fully creating a session will just take longer
Generally how long does this take? Is there any way to know when it’s done (in the code) ?
Another question is that, when the driver has not created all connections to all the shards of some node, is the driver already ready to be used to send requests to scylla server?