ScyllaDB is not starting on a private subnet server

ScyllaDB
1

I am trying to setup 2 node cluster of ScyllaDB(version - 5.2.9-0.20230920.5709d0043978) on AWS EC2 Server (Ubuntu) which is in a private subnet.
Node 1 IP - 10.0.137.63
Node 2 IP - 10.0.156.228

scylla.yaml →

cluster_name: ‘Test Cluster’
num_tokens: 256
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
schema_commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: “10.0.137.63”
listen_address: “10.0.137.63”
broadcast_address: “10.0.137.63”
native_transport_port: 9042
native_shard_aware_transport_port: 19042
read_request_timeout_in_ms: 5000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
endpoint_snitch: Ec2Snitch
rpc_address: localhost
rpc_port: 9160
api_port: 10000
api_address: “127.0.0.1”
batch_size_warn_threshold_in_kb: 128
batch_size_fail_threshold_in_kb: 1024
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
commitlog_total_space_in_mb: -1
murmur3_partitioner_ignore_msb_bits: 12
force_schema_commit_log: true
consistent_cluster_management: true
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/

Once I start the server I am getting below status

> root@ip-10-0-137-63:~# systemctl status scylla-server
> ● scylla-server.service - Scylla Server
>      Loaded: loaded (/lib/systemd/system/scylla-server.service; enabled; vendor preset: enabled)
>     Drop-In: /etc/systemd/system/scylla-server.service.d
>              └─capabilities.conf, dependencies.conf, sysconfdir.conf
>      Active: active (running) since Tue 2023-10-03 17:43:31 UTC; 10min ago
>     Process: 1552 ExecStartPre=/opt/scylladb/scripts/scylla_prepare (code=exited, status=0/SUCCESS)
>    Main PID: 1603 (scylla)
>      Status: "starting sstables loader"
>       Tasks: 4 (limit: 4667)
>      Memory: 96.9M
>         CPU: 1min 36.551s
>      CGroup: /scylla.slice/scylla-server.slice/scylla-server.service
>              └─1603 /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --io-properties->
> 
>  
> 
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system_schema.indexes 5fb8f950-6214-11ee-94d4-8a3545094fd>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system_schema.scylla_aggregates 5fd274c0-6214-11ee-94d4-8>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system_schema.scylla_aggregates 5fd274c0-6214-11ee-94d4-8>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system_schema.scylla_keyspaces 5fecb380-6214-11ee-94d4-8a>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system_schema.scylla_keyspaces 5fecb380-6214-11ee-94d4-8a>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system.local 600c2260-6214-11ee-94d4-8a3545094fd8] Compac>
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] gossip - failure_detector_loop: Started main loop
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] raft_group0 - setup_group0: joining group 0...
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] raft_group0 - server 5a7ac5ee-8f20-45b1-8a4b-3031dfe08de3 found no local group >
> Oct 03 17:43:34 ip-10-0-137-63 scylla[1603]:  [shard 0] compaction - [Compact system.local 600c2260-6214-11ee-94d4-8a3545094fd8] Compac>

On running cmd nodetool status, getting below output

> root@ip-10-0-137-63:~# nodetool status
> nodetool: Scylla API server HTTP GET to URL '/storage_service/ownership/' failed: runtime_exception (runtime error: No nodes present in the cluster. Has this node finished starting up?)
> See 'nodetool help' or 'nodetool help <command>'.
> Seems like the node is not able to connect to the cluster.

Here are few thing which I have already tried:

  1. check security group of Ec2 server. For now I have allowed ALL Traffic rule.
  2. Separately ran cmd on ubuntu server to open all the required ports
  3. Executed same steps on a public subnet server and everything is working fine.

Can anyone help me what could I do to resolve it.
Please let me know if any other information is required from my end.

2

It is not clear what the problem may be as the scylla.yaml you provided shows you are trying to bootstrap the first node of the cluster (a seed of itself), whereas the systemd output indicate bootstrapping is taking a long (10 min) period, which of course explains why your nodetool output is failing.

As you’re using Ec2Snitch, ensure you can reach AWS Metadata server. Then, update your scylla.yaml and properly set your rpc_address for CQL connections to listen outside of loopback only.

Finally, follow the Node Cleanup Procedure and try to bootstrap it again.