# Nodes not joining a cluster, incrementally adding nodes to a cluster **URL:** \ **Category:** ScyllaDB\ **Tags:** nodetool\ **Created:** [July 14, 2024, 1:38pm UTC](https://forum.scylladb.com/t/nodes-not-joining-a-cluster-incrementally-adding-nodes-to-a-cluster/2299 "2024-07-14T13:38:48Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [July 14, 2024, 1:38pm UTC](https://forum.scylladb.com/t/nodes-not-joining-a-cluster-incrementally-adding-nodes-to-a-cluster/2299/1 "2024-07-14T13:38:48Z")
> [@](#): > > [**Originally from the User Slack**](https://scylladb-users.slack.com/) > > ![Pham_William](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/a/aaeec8ae3263d3caffa867c8f39a6c95c2a713a5.png) **@Pham\_William:** Hey everyone, setting up a test cluster in a virtual environment, and I have a few questions. > Based on the configuration guide for a single data center: [https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/create-cluster.html](https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/create-cluster.html) > > I just have 2 nodes: > | scylladb | RUNNING | 10.70.188.92 (eth0) | fd42:ab78:d54:2140:216:3eff:fe59:33a7 (eth0) | CONTAINER | 0 | > ±----------±--------±---------------------±---------------------------------------------±----------------±----------+ > | scylladb2 | RUNNING | 10.70.188.39 (eth0) | fd42:ab78:d54:2140:216:3eff:feeb:2be7 (eth0) | CONTAINER | 0 | > > scylladb (10.70.188.92 - seed): > > ```auto > root@scylladb:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep seeds > - seeds: "10.70.188.92" > root@scylladb:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep rpc_address > rpc_address: 10.70.188.92 > root@scylladb:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep listen_address > listen_address: 10.70.188.92 > root@scylladb:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep endpoint_snitch > endpoint_snitch: GossipingPropertyFileSnitch > root@scylladb:~# cat /etc/scylla/cassandra-rackdc.properties > prefer_local=true > dc=datacenter1 > rack=rack1 > > ``` > > scylladb2 (10.70.188.39): > > ```auto > root@scylladb2:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep seeds > - seeds: "10.70.188.92" > root@scylladb2:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep rpc_address > rpc_address: 10.70.188.39 > root@scylladb2:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep listen_address > listen_address: 10.70.188.39 > root@scylladb2:~/server_dotfiles# cat /etc/scylla/scylla.yaml|grep endpoint_snitch > endpoint_snitch: GossipingPropertyFileSnitch > root@scylladb2:cat /etc/scylla/cassandra-rackdc.properties > prefer_local=true > dc=datacenter1 > rack=rack2 > > ``` > > I tried to confirm the client can connect to it, so using `cqlsh` , I got `Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': ConnectionRefusedError(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})`. `cqlsh --help` then suggested I to change the host with `$CQLSH_HOST`. So I did: > scylladb2 (10.70.188.39): > > ```auto > root@scylladb2:~# CQLSH_HOST=10.70.188.39 cqlsh > Connected to scylladb1 at 10.70.188.39:9042 > [cqlsh 6.0.18 | Scylla 6.0.0-0.20240606.a77615adf324 | CQL spec 3.3.1 | Native protocol v4] > Use HELP for help. > cqlsh> > > ``` > > I also tried to connect to node 1 with: > scylladb2 (10.70.188.39): > > ```auto > root@scylladb2:~# CQLSH_HOST=10.70.188.92 cqlsh > Connected to scylladb1 at 10.70.188.92:9042 > [cqlsh 6.0.18 | Scylla 6.0.0-0.20240606.a77615adf324 | CQL spec 3.3.1 | Native protocol v4] > Use HELP for help. > cqlsh> > > ``` > > And the documentation says I should run `nodetool status` to check the status of the cluster. But it failed with `error running operation: std::system_error (error system:111, Connection refused)` . So I checked the config which you can specify `-h` for host, so I did: > scylladb2 (10.70.188.39): > > ```auto > root@scylladb2:~# nodetool -h 10.70.188.39 status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > UN 10.70.188.39 2.20 MB 256 ? 353babac-b23e-4cf3-9bab-d09e9fa3ebd5 rack2 > > ``` > > This is a little bit odd for me because I set multiple nodes, but there’s only one node there. I can connect to node 1 as well, but I can only see one node there. > scylladb2 (10.70.188.39): > > ```auto > Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless > root@scylladb2:~# nodetool -h 10.70.188.92 status > Datacenter: datacenter1 > ======================= > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > UN 10.70.188.92 2.32 MB 256 ? b054e62c-8c89-4107-b839-1560b67988d4 rack1 > > Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless > > ``` > > My questions are: > > 1. Why is it not showing 2 nodes even though I set both the config for node 1 and node 2 to use node 1 as seed? > 2. In the documentation, `cqlsh` or `nodetool status` will automatically connect to the cluster. How does that work? In my case, I have to specify the IP address of one particular node, or it won’t connect. > 3. This is more like a subset of question 2: If I use the Rust library as a client to connect to scylladb, which IP address should I use for the cluster? Should it be the seed’s IP? > > > Create a ScyllaDB Cluster - Single Datacenter (DC) | ScyllaDB Docs > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) **@Felipe\_Cardeneti\_Mendes:** 1. It seems like node2 bootstrapped as a separate cluster and is unaware of node1. [Wipe it](https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/clear-data.html) clean and retry. > 2. Set `rpc_address` to 0.0.0.0, and `rpc_broadcast_address` to the node relevant IP > 3. You should specify the IP you set as RPC address as a contact point > > ![Pham_William](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/a/aaeec8ae3263d3caffa867c8f39a6c95c2a713a5.png) **@Pham\_William:** @Felipe\_Cardeneti\_Mendes That works! Thank you very much. I would assume I should be using the RPC address of the seed? Or it can be whatever node? > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) **@Felipe\_Cardeneti\_Mendes:** Any node is fine if you refer to the contact point for the app, but you more often than not want to specify more than a single node as a contact point > > ![Pham_William](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/a/aaeec8ae3263d3caffa867c8f39a6c95c2a713a5.png) **@Pham\_William:** Hi, @Felipe\_Cardeneti\_Mendes, there was some issue, so I started from scratch. With 3 nodes as follows: > ±----------±--------±---------------------±---------------------------------------------±----------------±----------+ > | scylladb1 | RUNNING | 10.70.188.63 (eth0) | fd42:ab78:d54:2140:216:3eff:fef1:2d6a (eth0) | CONTAINER | 0 | > ±----------±--------±---------------------±---------------------------------------------±----------------±----------+ > | scylladb2 | RUNNING | 10.70.188.242 (eth0) | fd42:ab78:d54:2140:216:3eff:fe67:72ef (eth0) | CONTAINER | 0 | > ±----------±--------±---------------------±---------------------------------------------±----------------±----------+ > | scylladb3 | RUNNING | 10.70.188.145 (eth0) | fd42:ab78:d54:2140:216:3eff:fea9:4149 (eth0) | CONTAINER | 0 | > ±----------±--------±---------------------±---------------------------------------------±----------------±----------+ > > ```auto > - node 1 > cluster_name: 'scylladb' > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "10.70.188.63" > listen_address: 10.70.188.63 > endpoint_snitch: GossipingPropertyFileSnitch > rpc_address: 0.0.0.0 > api_address: 10.70.188.63 > broadcast_rpc_address: 10.70.188.63 > > - node 2 > cluster_name: 'scylladb' > listen_address: 10.70.188.242 > api_address: 10.70.188.242 > rpc_address: 0.0.0.0 > broadcast_rpc_address: 10.70.188.242 > endpoint_snitch: GossipingPropertyFileSnitch > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "10.70.188.63" > > - node 3 > cluster_name: 'scylladb' > listen_address: 10.70.188.145 > api_address: 10.70.188.145 > rpc_address: 0.0.0.0 > broadcast_rpc_address: 10.70.188.145 > endpoint_snitch: GossipingPropertyFileSnitch > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "10.70.188.63" > > ``` > > They all have the same /etc/scylla/cassandra-rackdc.properties (only difference in rack number 1,2,3) > > ```auto > dc=sydney > rack=rack1 > prefer_local=true > > ``` > > Now, if I run nodetool status on whatever node, will all return node 1 status only. > > ```auto > root@scylladb1:~# nodetool -h 10.70.188.63 status > Datacenter: sydney > ================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > UN 10.70.188.63 1.99 MB 256 ? 410f137f-d213-4cd3-8729-976b1101464c rack1 > > Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless > > ``` > > node 2 and 3 will show node 1 only but without information on its load > > ```auto > > root@scylladb2:~# nodetool -h 10.70.188.242 status > root@scylladb3:~# nodetool -h 10.70.188.145 status > Datacenter: sydney > ================== > Status=Up/Down > |/ State=Normal/Leaving/Joining/Moving > -- Address Load Tokens Owns Host ID Rack > DN 10.70.188.63 ? 256 ? 410f137f-d213-4cd3-8729-976b1101464c rack1 > > Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless > > ``` > > I have also tried to remove the data from nodes 2 and 3, but no luck. Is there a reason why it’s only showing node 1 with status DN “Down Normal”, instead of up?: > > ```auto > bash > sudo rm -rf /var/lib/scylla/data > sudo find /var/lib/scylla/commitlog -type f -delete > sudo find /var/lib/scylla/hints -type f -delete > sudo find /var/lib/scylla/view_hints -type f -delete > > ``` > > Note that, systemctl status scylla-server all returns active (running) > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) **@Felipe\_Cardeneti\_Mendes:** Is this a “local” docker deployment? How are you making changes to the config file? In general, it’s best to start with a single node, then add nodes incrementally until you get the config and steps right. For example, if you add the 2nd node and it doesn’t work, then probably it bootstrapped early and you have to clear it up. Check its logs, see what happened, then correct whatever insights the logs might be telling you. Once you’ve got 2 nodes up, just repeat the same for other particular nodes ---
**Author:** ![ChaitDevOps](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/chaitdevops/32/994_2.png) [@ChaitDevOps](https://forum.scylladb.com/u/ChaitDevOps)\ **Post date:** [November 13, 2024, 5:26pm UTC](https://forum.scylladb.com/t/nodes-not-joining-a-cluster-incrementally-adding-nodes-to-a-cluster/2299/2 "2024-11-13T17:26:33Z")
set rpc\_address to 0.0.0.0 and broadcast\_rpc\_address to private\_ip. restart the cluster. this is how i solved it.