Installation details
#ScyllaDB version: 6.2 with tablets enabled
#Cluster size: 12 x 3dc
os (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu 24.10
Hi.
I have a ghost node
select * from system.cluster_status;
peer | dc | host_id | load | owns | status | tokens | up
--------+-------------+-------------+--------+----------+----------+--------+-------
10.0.0.1| null | null | | null | null | 0 | False
...
It does not appear in system.raft_state, but I got its host_id from nodetool gossipinfo.
Attempts to removenode it lead to
(removenode: node d6add466-... is in 'none' state. Wait for it to be in 'normal' state)
I can not assasinate it, because raft.
Rest api’s /gossiper/force_remove_endpoint/10.0.0.1 would happily report that it “Finished to force remove node” and do nothing (it would do so for 8.8.8.8 also).
I think it prevents getting my cluster in a working state by blocking topology ops.
As with my other problem I don’t have exact steps that led to this, one of them ran out of disk space and I’ve tried to replace it with the others.
What are my options?