Cqlsh Connection error: 'Unable to connect to any servers'

When I trying to connect to CQL Shell, I get an error below.
Anyone knows the resolution? Or how can I find out what’s wrong?

$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': ConnectionRefusedError(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Environment and background

  • OS: Oracle Linux 8.9
  • ScyllaDB: upgraded from 5.4.6 to 6.1.0. I’ve just re-installed 6.1.0 on 5.4.6 because I didn’t know how to uninstall ScyllaDB 5.4.6.
  • /etc/scylla/scylla.yaml: I changed ‘cluster_name’, ‘seeds’, ‘listen_address’, ‘endpoint_snitch’, ‘rcp_address’. The target machine should be a Seed Node.

What I tried

Started scylla-server and checked the status.

$ sudo systemctl start scylla-server
$ sudo systemctl status scylla-server
● scylla-server.service - Scylla Server
   Loaded: loaded (/usr/lib/systemd/system/scylla-server.service; disabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/scylla-server.service.d
           mqcapabilities.conf, dependencies.conf
   Active: active (running) since Wed 2024-08-21 18:07:14 JST; 2s ago
  Process: 595323 ExecStartPre=/opt/scylladb/scripts/scylla_prepare (code=exited, status=0/SUCCESS)
 Main PID: 595331 (scylla)
   Status: "serving"
    Tasks: 8 (limit: 97456)
   Memory: 420.2M
   CGroup: /scylla.slice/scylla-server.slice/scylla-server.service
           mq595331 /usr/bin/scylla --log-to-syslog 1 --log-to-stdout 0 --default-log-level info --network-stack posix --smp 4 --io-properties-file=/etc/scylla.d/io_properties.y>

 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.views c22905e0-5f9c-11ef-bc51-045f240980d1] Compacted 2 sstables to [/var/lib/scy>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.view_virtual_columns c22a6570-5f9c-11ef-bc51-045f240980d1] Compacting [/var/lib/s>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] sstable - Rebuilding bloom filter /var/lib/scylla/data/system_schema/view_virtual_columns-08843b6345dc3be29798a041829>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.view_virtual_columns c22a6570-5f9c-11ef-bc51-045f240980d1] Compacted 2 sstables t>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.scylla_tables c22bc500-5f9c-11ef-bc51-045f240980d1] Compacting [/var/lib/scylla/d>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] sstable - Rebuilding bloom filter /var/lib/scylla/data/system_schema/scylla_tables-5d912ff1f7593665b2c88042ab5103dd/m>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.scylla_tables c22bc500-5f9c-11ef-bc51-045f240980d1] Compacted 2 sstables to [/var>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.scylla_aggregates c22d2490-5f9c-11ef-bc51-045f240980d1] Compacting [/var/lib/scyl>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] sstable - Rebuilding bloom filter /var/lib/scylla/data/system_schema/scylla_aggregates-08d8cb2892023371a968ad926e0fdc>
 8月 21 18:07:14 pc-fmvlb47 scylla[595331]:  [shard 0:comp] compaction - [Compact system_schema.scylla_aggregates c22d2490-5f9c-11ef-bc51-045f240980d1] Compacted 2 sstables to [>

Checked the Node status.

$ nodetool status
Datacenter: DC1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address       Load    Tokens Owns Host ID                              Rack
UN 172.16.33.250 1.17 MB 256    ?    2032ed85-3ee5-4e1c-9ff1-84978ea5e184 RAC1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless

Tried to connect CQL shell. Then, connection error occurred. :face_with_spiral_eyes:

$ cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042': ConnectionRefusedError(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

I tried to show the TCP port condition.
Indeed, port 9042 is being listened, but it’s different IP than the one the process is trying to connect to with cqlsh.
I could connect to CQL shell when I used the ScyllaDB v5.4.6, though I configured no special settings to connect to CQL shell. What’s different or wrong now?

$ ss -atn
State    Recv-Q   Send-Q       Local Address:Port         Peer Address:Port    Process
LISTEN   0        244              127.0.0.1:5432              0.0.0.0:*
LISTEN   0        100          172.12.34.567:7000              0.0.0.0:*
LISTEN   0        64                 0.0.0.0:42709             0.0.0.0:*
LISTEN   0        100          172.12.34.567:19042             0.0.0.0:*
LISTEN   0        100              127.0.0.1:10000             0.0.0.0:*
LISTEN   0        128                0.0.0.0:33695             0.0.0.0:*
LISTEN   0        50                 0.0.0.0:445               0.0.0.0:*
LISTEN   0        128                0.0.0.0:22                0.0.0.0:*
LISTEN   0        128                0.0.0.0:111               0.0.0.0:*
LISTEN   0        50                 0.0.0.0:139               0.0.0.0:*
LISTEN   0        128                0.0.0.0:20048             0.0.0.0:*
LISTEN   0        100          172.12.34.567:9042              0.0.0.0:*  <--
LISTEN   0        100          172.12.34.567:9180              0.0.0.0:*
LISTEN   0        64                 0.0.0.0:2049              0.0.0.0:*
ESTAB    0        0            172.12.34.567:51598          172.xx.x.x:445
ESTAB    0        0            172.12.34.567:47254         172.xx.x.xx:49164
ESTAB    0        0            172.12.34.567:49604       172.12.34.567:7000
ESTAB    0        0            172.12.34.567:22          172.xx.xx.xxx:57915
ESTAB    0        0            172.12.34.567:7000        172.12.34.567:49604
ESTAB    0        36           172.12.34.567:22          172.xx.xx.xxx:56425
LISTEN   0        244                  [::1]:5432                 [::]:*
LISTEN   0        128                   [::]:47357                [::]:*
LISTEN   0        4096                     *:9100                    *:*
LISTEN   0        50                    [::]:445                  [::]:*
LISTEN   0        128                      *:23                      *:*
LISTEN   0        128                   [::]:22                   [::]:*
LISTEN   0        32                       *:21                      *:*
LISTEN   0        128                   [::]:111                  [::]:*
LISTEN   0        50                    [::]:139                  [::]:*
LISTEN   0        128                   [::]:20048                [::]:*
LISTEN   0        64                    [::]:35343                [::]:*
LISTEN   0        64                    [::]:2049                 [::]:*

what version of CQLsh are you using?
Maybe try to upgrade to the latest GitHub - scylladb/scylla-cqlsh: A fork of the cqlsh code

Scylla will use the address specified in rpc_address, as the address to listen on, for CQL connections. You mention above that you have changed this configuration, this is probably why your ScyllaDB node now doesn’t listen on the (default) 127.0.0.1 address anymore.

Also note that cqlsh, when launched without arguments, will attempt to connect to localhost (127.0.0.1). If your ScyllaDB node listens on another address, you can provide this as a positional command-line argument:

$ cqlsh 172.12.34.567

Thanks for the suggestion.
I got the following log in installing. I thought it was enough updated, but it’s not enough?

Installed:
  scylla-6.1.0-0.20240804.abbf0b24a60c.x86_64
  scylla-conf-6.1.0-0.20240804.abbf0b24a60c.x86_64
  scylla-cqlsh-6.1.0-0.20240804.abbf0b24a60c.x86_64    <--here!!
  scylla-kernel-conf-6.1.0-0.20240804.abbf0b24a60c.x86_64
  scylla-node-exporter-6.1.0-0.20240804.abbf0b24a60c.x86_64
  scylla-python3-6.1.0-0.20240804.abbf0b24a60c.x86_64
  scylla-server-6.1.0-0.20240804.abbf0b24a60c.x86_64

Anyway, I will keep in mind that I can upgrade it individually from there.
Thanks for the valuable information!

Thanks for the info!
Yeah, I also thought maybe that changing the settings in scylla.yaml would affect it. I got that the setting of rpc_address is also reflected in the listen address of cqlsh.
And I didn’t know you could specify arguments to cqlsh command. I’ll give it a try!
Thanks alot.

I tried to this way, but doesn’t work. I got the same error as before.

I’m so glad to let me know anything If you guys have another idea.
Thank you!

I copied from your log output the address which seemed to be the one on which CQL listens. But I may have copied the wrong address. You can find out the correct address yourself: when a ScyllaDB node starts up, it will print the address it listens on:

INFO  2024-09-18 00:20:14,142 [shard 0:stmt] cql_server_controller - Starting listening for CQL clients on 127.0.0.1:9042 (unencrypted, non-shard-aware)                                                                                                       
INFO  2024-09-18 00:20:14,142 [shard 0:stmt] cql_server_controller - Starting listening for CQL clients on 127.0.0.1:19042 (unencrypted, shard-aware)  

This is the address you have to pass to cqlsh and other clients. Note that you have to make sure that this address is one that is reachable from where your client is running. if you are running ScyllaDB in a docker container, you might have to open the appropriate ports.