Raft server id ... cannot be translated to an IP address

ScyllaDB
, ,
1

Hi,

yesterday I updated a cluster to Scylla 6.0.2.

I am seeing quite a lot of these errors:

Aug 12 07:12:47 osdev-6 scylla[3776943]:  [shard  2:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id 8a627941-2f40-47ad-8e5d-6f6e891ab85d cannot be translated to an IP address.
Aug 12 07:12:49 osdev-6 scylla[3776943]:  [shard  7:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id e16a9c96-d8a0-47fe-8044-37be077f45b9 cannot be translated to an IP address.
Aug 12 07:12:50 osdev-6 scylla[3776943]:  [shard  6:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id d728fc9d-81ca-4f34-ab5b-3b0858144c61 cannot be translated to an IP address.

Aug 12 08:07:55 osdev-2 scylla[4101879]:  [shard  7:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id 8a627941-2f40-47ad-8e5d-6f6e891ab85d cannot be translated to an IP address.
Aug 12 08:07:55 osdev-2 scylla[4101879]:  [shard  1:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id e16a9c96-d8a0-47fe-8044-37be077f45b9 cannot be translated to an IP address.
Aug 12 08:07:55 osdev-2 scylla[4101879]:  [shard  3:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id d728fc9d-81ca-4f34-ab5b-3b0858144c61 cannot be translated to an IP address.
Aug 12 08:12:55 osdev-2 scylla[4101879]:  [shard  7:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id 8a627941-2f40-47ad-8e5d-6f6e891ab85d cannot be translated to an IP address.
Aug 12 08:12:55 osdev-2 scylla[4101879]:  [shard  1:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id e16a9c96-d8a0-47fe-8044-37be077f45b9 cannot be translated to an IP address.
Aug 12 08:12:55 osdev-2 scylla[4101879]:  [shard  3:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id d728fc9d-81ca-4f34-ab5b-3b0858144c61 cannot be translated to an IP address.
Aug 12 08:17:55 osdev-2 scylla[4101879]:  [shard  7:main] raft_group_registry - (rate limiting dropped 2997 similar messages) Raft server id 8a627941-2f40-47ad-8e5d-6f6e891ab85d cannot be translated to an IP address.
Aug 12 08:17:55 osdev-2 scylla[4101879]:  [shard  1:main] raft_group_registry - (rate limiting dropped 2998 similar messages) Raft server id e16a9c96-d8a0-47fe-8044-37be077f45b9 cannot be translated to an IP address.
Aug 12 08:17:55 osdev-2 scylla[4101879]:  [shard  3:main] raft_group_registry - (rate limiting dropped 2998 similar messages) Raft server id d728fc9d-81ca-4f34-ab5b-3b0858144c61 cannot be translated to an IP address.

They appear in bursts. It even seems like queries are being affected by this as I had my application hang while this was happening, but I need more investigation to be sure about this.

Is this normal or might there be a bug?

Is there really no way to return a cluster to gossip?

regards,
Christian

2

For my understanding:

If Raft is down, are only schema/topology changes down, or also read/writes?

3

As per https://www.scylladb.com/2023/05/04/scylladbs-path-to-strong-consistency-a-new-milestone/:slight_smile :

" If a node is partitioned away from the cluster, it can’t perform schema changes. That’s the main difference, or limitation, from the pre-Raft clusters that you should keep in mind. You can still perform other operations with such nodes (such as reads and writes) so data availability is unaffected."

I am relieved to read this. There are still some things to test, e.g.: What if a DC is nuked and never comes back. Will I be able to forcefully remove the nodes from the nuked DC? Otherwise there is no way to ever recover from this.

Would be nice if there was some kind of force option.