# Nodes stuck in start up phase, seed ip and listen address **URL:** \ **Category:** ScyllaDB\ **Tags:** troubleshooting, configuration\ **Created:** [July 16, 2024, 4:42am UTC](https://forum.scylladb.com/t/nodes-stuck-in-start-up-phase-seed-ip-and-listen-address/2309 "2024-07-16T04:42:08Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [July 16, 2024, 4:42am UTC](https://forum.scylladb.com/t/nodes-stuck-in-start-up-phase-seed-ip-and-listen-address/2309/1 "2024-07-16T04:42:08Z")
> [@](#): > > [**Originally from the User Slack**](https://scylladb-users.slack.com/) > > ![Joakim_Lindqvist](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/8/8353eed1eb45b8171f3a3fbb68432f1869224b1d.jpeg) **@Joakim\_Lindqvist:** Hey > > I am attempting to test scylla 6.0.1 on a new cluster and running into an issue were my nodes seems stuck starting up in phase `starting the view builder`. > 🧵 > This node that is starting up has ip `192.168.2.239` and is set to be the seed node. > This is on AWS using the scylla amis in us-east-1, using `i4i.large` instances. This is meant as a small dev deployment thus the small scale. > > The scylla monitoring is reporting the status as `Starting`. I have given this node about 2 hours and it still hasn’t moved. > Scylla.yaml > > ```auto > # Generated by Scylla Machine Image at 2024-06-25 09:30:46.129293 > # See '/etc/scylla/scylla.yaml.example' with the full list of supported configuration > # options and their descriptions. > api_address: 127.0.0.1 > api_doc_dir: /opt/scylladb/api/api-doc/ > api_port: 10000 > api_ui_dir: /opt/scylladb/swagger-ui/dist/ > auto_bootstrap: true > batch_size_fail_threshold_in_kb: 1024 > batch_size_warn_threshold_in_kb: 128 > broadcast_rpc_address: 192.168.2.239 > cas_contention_timeout_in_ms: 1000 > cluster_name: jupiter-dev > commitlog_segment_size_in_mb: 32 > commitlog_sync: periodic > commitlog_sync_period_in_ms: 10000 > commitlog_total_space_in_mb: -1 > enable_tablets: true > endpoint_snitch: GossipingPropertyFileSnitch > listen_address: 192.168.2.239 > maintenance_socket: ignore > murmur3_partitioner_ignore_msb_bits: 12 > native_shard_aware_transport_port: 19042 > native_transport_port: 9042 > num_tokens: 256 > partitioner: org.apache.cassandra.dht.Murmur3Partitioner > read_request_timeout_in_ms: 5000 > rpc_address: 0.0.0.0 > rpc_port: 9160 > schema_commitlog_segment_size_in_mb: 128 > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "192.168.2.239" > strict_is_not_null_in_views: true > write_request_timeout_in_ms: 2000 > > ``` > > cassandra-rackdc.properties > > ```auto > prefer_local=true > dc=us-east-1 > rack=us-east-1b > > ``` > > Journal log > > ![dor](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/27b698a3e588f84cb82a722e458d31c997a48bed.jpeg) **@dor:** I am not sure but maybe it’s related to Raft not finding a majority, as it’s a single node cluster. It should work but maybe it’s an issue. If you’ll add 2 more nodes, you’ll see whether I was right or not > > ![Joakim_Lindqvist](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/8/8353eed1eb45b8171f3a3fbb68432f1869224b1d.jpeg) **@Joakim\_Lindqvist:** I attempted to add one more node and that didn’t seem to help. I can add another one to see if it fixes itself. > The 3rd node is up, doesn’t seem to be making progress. > > ![dor](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/27b698a3e588f84cb82a722e458d31c997a48bed.jpeg) **@dor:** hmm, sorry, I didn’t see anything special in the logs. I assume the config is completely fresh (minus edits) > > ![Joakim_Lindqvist](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/8/8353eed1eb45b8171f3a3fbb68432f1869224b1d.jpeg) **@Joakim\_Lindqvist:** Yeah the config is all generated by the machine image except for my edits to the seed ip > > ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** @Kamil\_Braun please help > > **@Kamil\_Braun:** \> I am not sure but maybe it’s related to Raft not finding a majority, as it’s a single node cluster. > “majority” is always calculated from total number of nodes. If your total is 1 then your majority is 1. If total is 3 then majority is 2 etc. > So single node clusters work as usual. > I suspect that you tried to first boot the node with a different seed by mistake, then stopped it and changed the seed. But the old seed was persisted in the cluster discovery algorithm which now requires to contact that seed in order to proceed > OR you concurrently tried to start a second node which contacted this node and inserted itself as one of participants of cluster discovery algorithm, and then shut down. But it’s also one of the required contact points now. > In any case the node looks to be stuck inside cluster discovery algorithm. > stop it, delete everything (workdir i.e. data, commitlog, etc.), make sure you have correct seed setup (see documentation), try again > > ![Joakim_Lindqvist](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/8/8353eed1eb45b8171f3a3fbb68432f1869224b1d.jpeg) **@Joakim\_Lindqvist:** I did initially start it with `127.0.0.1` as the seed ip, yes. > I did start the second node after a while (as I suspected a similar issue to Dor were we might need more nodes for it to start) but it should have started up in time for that. > > I will tear it all down and try again and see if I can repro it. Thanks for the suggestion! > I have tested tearing this down and setting up a new cluster. This time with a single node, this is set to have `127.0.0.1` as its seed node (seems to be what happens by default when I do not specify any seed ips). > There are no other nodes attempting to connect. > > I am still seeing the same issue with it stopping at the same point. > > **@Kamil\_Braun:** You cannot use `127.0.0.1` if your `listen_address` / `broadcast_address` etc. is different > you must use the same IP everywhere > [https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/create-cluster.html](https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/create-cluster.html) > > > Create a ScyllaDB Cluster - Single Datacenter (DC) | ScyllaDB Docs > > so if you used `listen_address: 192.168.2.239` > then you must use > > ```auto > > seed_provider: > - class_name: org.apache.cassandra.locator.SimpleSeedProvider > parameters: > - seeds: "192.168.2.239" > > ``` > > but now you have to teardown again. If you use `seeds: "127.0.0.1"` and then change later to `seeds: "192.168.2.239"` it won’t work because the cluster discovery algorithm already persisted `127.0.0.1` and will continue trying to contact it (and never finish) > > ![Joakim_Lindqvist](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/8/8353eed1eb45b8171f3a3fbb68432f1869224b1d.jpeg) **@Joakim\_Lindqvist:** I didn’t explicitly set 127.0.0.1 as the seed or listen address, I set the seed to empty string which seems to then default to that ip. > The machine image generated yaml sets the listen address to the ip of the machine. > > But I guess this may have been a result of my changes to how our bootstrapping works were I do not allocate a ec2 ip ahead of time for our machines and instead try to update it after the machine has started and the ip has been allocated. But I am fairly sure we have been able to setup new clusters like this in 5.x (we never tried this with 4.x as we had predefined ips for those). > > But this gives me some data to go on, I will try and see if I can rework our bootstrapping to work better without having to rely on predefined ips (as that results in special case handling of this initial seed node which doesn’t really make sense with how seeds are not really a thing anymore once the cluster starts up). > I resolved this now. Avoiding specifying a seed ip for the first node, e.g. a machine image user data like this: > > ```auto > { > "scylla_yaml": { > "cluster_name": "jupiter-dev", > "endpoint_snitch": "GossipingPropertyFileSnitch" > }, > "post_configuration_script" : "" > } > > ``` > > Resolves this issue as the machine image defaults to the private ip of the ec2 instance. > > So this was really a problem in our ec2 setup, I am unclear on why this was not a issue for the new clusters we have setup with 5.x but fundamentally we were not following the setup recommendations and doing so resolves the issue.