(rate limiting dropped 498 similar messages) Waiting for 2 live nodes to show up in gossip, currently 1 present

ScyllaDB
, , ,
1

Installation details
#ScyllaDB version: 5.4.9
#Cluster size: 3
os (RHEL/CentOS/Ubuntu/AWS AMI): AWS AMI

I have a cluster having 3 nodes. Out of them 2 are seed node. I am trying to add 1 more node. Going forward this node is going to be the seed node. We would obsolete existing 3 nodes.
What would be the correct strategy for this.

Currently I am trying to add 1 new node to existing 2 seed node by giving seed as xx.5.2.1,xx.5.1.1

I am getting an error shard 0:main] init - Startup failed: std::runtime_error (Timed out waiting for 2 live nodes to show up in gossip)

I have checked 7000,9042 ports. Its connecting from the new node. Can anybody help in this?

2

Hi @Amit_Sahoo

I would revalidate the configuration on the new node -

cluster_name: <existing_cluster_name>  # Must match all nodes
seeds: "<seed_ip_1>,<seed_ip_2>"       # Verify exact IPs
listen_address: <new_node_private_ip>   # Must be reachable
rpc_address: <new_node_private_ip>      # Must be reachable

on an existing node - check the nodes health, using nodetool status; all nodes should show “UN”

As for the replacement strategy itself -

  1. any topology change will perform rebalacing of the tokes and therefore the data, keep that in mind; I suggest first to add the new node and then replace the old-ones.
  2. I read you are planning to add one node and remove 2, it means you will be left with 2 nodes. make sure the replication factor in your keyspaces is not higher than 2. I would suggest to go with a 3 node RF=3 strategy for better redundancy and use LOCAL_QUORUM for read and writes.

Good luck,
Gabriel.

1 Like