# Stuck topology requests **URL:** \ **Category:** ScyllaDB\ **Tags:** troubleshooting, administration, topology\ **Created:** [May 26, 2025, 1:37pm UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856 "2025-05-26T13:37:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Andrey\_Kojevnikov](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/andrey_kojevnikov/32/1548_2.png) [@Andrey\_Kojevnikov](https://forum.scylladb.com/u/Andrey_Kojevnikov)\ **Post date:** [May 26, 2025, 1:37pm UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/1 "2025-05-26T13:37:16Z")
_Installation details_ #ScyllaDB version: 6.2 #Cluster size: 12 nodes x 3 dc #OS: Ubuntu 24.10 Hi. I had issues with couple of nodes which led to them being down, unable to be removed and replaced. ```auto SELECT * FROM system.topology_requests WHERE done = False ALLOW FILTERING ; id | done | end_time | error | initiating_host | request_type | start_time --------------+-------+----------+-------+-----------------+--------------+----------- cd9b84c0-... | False | null | null | 59853b48-... | replace | null 524e7f0c-... | False | null | null | a48691bf-... | remove | null 8a5185e6-... | False | null | null | a48691bf-... | remove | null acc4f340-... | False | null | null | 094374c3-... | replace | null ``` I don’t have exact steps that led to this, one of them ran out of disk space and I’ve tried to replace it with the other. As I understand they are blocking other cluster ops, and removing records from topology\_requests table will not help. How can i remove/cancel stuck topology requests? ---
**Author:** ![Andrey\_Kojevnikov](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/andrey_kojevnikov/32/1548_2.png) [@Andrey\_Kojevnikov](https://forum.scylladb.com/u/Andrey_Kojevnikov)\ **Post date:** [May 26, 2025, 1:41pm UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/2 "2025-05-26T13:41:30Z")
Also I have ks with tablets enabled, so no manual recovery. ---
**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\ **Post date:** [June 11, 2025, 6:25am UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/3 "2025-06-11T06:25:53Z")
It would help if you attached logs. I have two guesses from your description: 1. if you have more than one node which is lost, you have to call removenode/replace on all of them (in parallel, not waiting for completion). This is so that those nodes are marked as “permanently dead” and topology operations don’t try to synchronize with them. If there is a regular node which is down, topology operations will be stuck trying to synchronize with them. 2. if you lost majority of nodes, group0 quorum is lost, and topology operations will be stuck on commits to raft. In that case, you have to perform group0 recovery procedure first. ---
**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\ **Post date:** [June 11, 2025, 6:28am UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/4 "2025-06-11T06:28:40Z")
Before logs, the contents of system.topology and system.cluster\_status would also help. ---
**Author:** ![Andrey\_Kojevnikov](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/andrey_kojevnikov/32/1548_2.png) [@Andrey\_Kojevnikov](https://forum.scylladb.com/u/Andrey_Kojevnikov)\ **Post date:** [June 12, 2025, 9:14am UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/5 "2025-06-12T09:14:15Z")
Hi. Thank You for the reply. two nodes are down, yes I don’t think removenode can take more than one host id removenode was executed with --ignore-dead-nodes for both subsequents attempts result in “Concurrent request for removal already in progress” for both no, quorum is not lost, i have 12 nodes per each of 3 dc does restoring group0 implies “manual recovery”? because per documentation it is incompatible with tablets being enabled I also have a ghost node, which might be the cause of this: [Ghost node in none state](https://forum.scylladb.com/t/ghost-node-in-none-state/4875) ---
**Author:** ![Andrey\_Kojevnikov](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/andrey_kojevnikov/32/1548_2.png) [@Andrey\_Kojevnikov](https://forum.scylladb.com/u/Andrey_Kojevnikov)\ **Post date:** [June 12, 2025, 9:20am UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/6 "2025-06-12T09:20:26Z")
I have ungodly amount of logs. What should I look for and on which node? I have yet to find anything of practical value. ---
**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\ **Post date:** [June 14, 2025, 2:13pm UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/7 "2025-06-14T14:13:17Z")
Maybe we don’t need logs. First, let’s check the status of topology. On one of the nodes: select \* from system.cluster\_status; select \* from system.topology; ---
**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\ **Post date:** [June 14, 2025, 2:29pm UTC](https://forum.scylladb.com/t/stuck-topology-requests/4856/8 "2025-06-14T14:29:04Z")
> removenode was executed with --ignore-dead-nodes for both > subsequents attempts result in “Concurrent request for removal already in progress” for both That’s fine, the nodes should still be marked as excluded (can be observed in system.topology in the “ignore\_nodes” column). > no, quorum is not lost, i have 12 nodes per each of 3 dc > does restoring group0 implies “manual recovery”? > because per documentation it is incompatible with tablets being enabled Yes, but it’s not needed here. If you have a ghost node, then indeed it could block progress. You can manually fix it like this: 1. on every alive node, execute in cqlsh: delete from system.topology where key = ‘topology’ and host\_id = \; delete from system.peers where peer = \; 2. then do a rolling restart Observe if things start moving. If they are, the node on which topology coordinator runs should start logging progress. Leader runs on the node which was the last to log `raft_group0 - gaining leadership`. On topology updates, so also when tablet migration is moving forward, it will logs “raft\_topology - updating topology state:”