Why am I seeing more than expected gossip writes in a 6-node Scylla cluster? (26 writes instead of 18)

ScyllaDB
1

Installation details
#ScyllaDB version: 6.0.2
#Cluster size: 6
os (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI ami-0d0131dc0789683cc

Hello everyone,

I have a 6-node Scylla cluster, and according to the official Scylla documentation, each node communicates with 1 to a maximum of 3 other nodes using the gossip protocol. However, when I check the gossip writes, I’m seeing about 26 writes instead of the expected 18, which corresponds to the 3x number of nodes in my cluster.

Has the number of gossip writes increased recently, or could there be another reason for this behavior?

Here are some additional details:
#ScyllaDB version: 6.0.2
#Cluster size: 6
os (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI ami-0d0131dc0789683cc

  • I’ve verified that all nodes are up and running without any obvious issues (e.g., no node failures or partitions).
  • I’m using the default gossip settings unless something has changed.

Any insights or pointers on why this is happening would be greatly appreciated!

Thanks in advance!

2

However, when I check the gossip writes, I’m seeing about 26 writes

How do you count gossip writes? metric, log, other?

3

I am able to see the gossip writes on scylla monitor in advanced tab.

4

Sample Images:

Screenshot 2024-12-18 at 11.36.55 AM
Screenshot 2024-12-18 at 11.36.55 AM1078×462 51 KB

5

Gossiper exchange is not one way, it exchanges information both ways, so theoretically, if each exchange result in a new information to both parties the max amount of writes should be 36. But gossiper does not suppose to write on each exchange, only if there is a new information that has to persisted (which is not usually the case), so the number in steady state should be 0.

The gossiper is not the only thing that runs is gossiper scheduling group though, so it is hard to say where the writes are coming from without digging much deeper.

1 Like