# Upgrade process stuck on build\_coordinator\_state - replication settings for system tables issue

**URL:** <https://forum.scylladb.com/t/upgrade-process-stuck-on-build-coordinator-state-replication-settings-for-system-tables-issue/2972>\
**Category:** ScyllaDB\
**Tags:** troubleshooting, raft, upgrade, replication\
**Created:** [September 30, 2024, 4:30am UTC](https://forum.scylladb.com/t/upgrade-process-stuck-on-build-coordinator-state-replication-settings-for-system-tables-issue/2972 "2024-09-30T04:30:22Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\
**Post date:** [September 30, 2024, 4:30am UTC](https://forum.scylladb.com/t/upgrade-process-stuck-on-build-coordinator-state-replication-settings-for-system-tables-issue/2972/1 "2024-09-30T04:30:22Z")

</div>

> [@](#):
>
> [**Originally from the User Slack**](https://scylladb-users.slack.com/)
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** Hi, I was trying to enable consistent topology updates in my cluster (scylla open source 6.0.3 everywhere, 3 dcs, 26 nodes) and it is stuck on `build_coordinator_state`. What might be a problem?
> 
> ![Piotr_Smaroń](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/c7e93f1f72e94abeeb7572b9785f007b82cc7308.png) **@Piotr\_Smaroń:** CC @Kamil\_Braun
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** I verified all prerequisites and started upgrade via curl ([http://127.0.0.1:10000/storage\_service/raft\_topology/upgrade](http://127.0.0.1:10000/storage_service/raft_topology/upgrade))  
> after migrating from 5.4.7 to 6.0.3.  
> I followed the procedure described in here: [https://opensource.docs.scylladb.com/branch-6.0/upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology.html](https://opensource.docs.scylladb.com/branch-6.0/upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology.html)
> 
> > Enable Consistent Topology Updates | ScyllaDB Docs
> 
> ![Kamil_Braun](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/16acd791289c6131034d6210e28a3368c9847a99.png) **@Kamil\_Braun:** \> What might be a problem?  
> Anything could be a problem. Without logs, it’s impossible to say.
> 
> cc @Piotr\_Dulikowski (author of the upgrade procedure)
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** sure. Plz let me know what to provide. I don’t see anything wrong in logs. Just
> 
> ```auto
> wrz 10 14:27:28 ams-scylla1 scylla[2458175]: [shard 0:strm] api - Requested to schedule upgrade to raft topology
> wrz 10 14:27:28 ams-scylla1 scylla[2458175]: [shard 0:strm] raft_topology - requesting to start upgrade to topology on raft
> wrz 10 14:27:28 ams-scylla1 scylla[2458175]: [shard 0:strm] raft_topology - upgrade to raft topology has started
> wrz 10 14:27:28 ams-scylla1 scylla[2458175]: [shard 0:strm] raft_topology - upgrade to topology on raft is scheduled
> 
> ```
> 
> ![Kamil_Braun](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/16acd791289c6131034d6210e28a3368c9847a99.png) **@Kamil\_Braun:** First try the recommendations from [https://opensource.docs.scylladb.com/branch-6.0/upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology.html#what[…]tuck](https://opensource.docs.scylladb.com/branch-6.0/upgrade/upgrade-opensource/upgrade-guide-from-5.4-to-6.0/enable-consistent-topology.html#what-if-upgrade-gets-stuck)
> 
> If nothing works (including rolling restart) – I recommend opening an issue and attaching logs **from all of your nodes** from the moment you started the upgrade procedure, ±1h
> 
> Also `nodetool status` output from one of the nodes
> 
> > Enable Consistent Topology Updates | ScyllaDB Docs
> 
> ![Piotr_Dulikowski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/0/059db617d110231ced9a984559a6da500b22c526.png) **@Piotr\_Dulikowski:** \> `upgrade to topology on raft is scheduled`  
> After this, the topology coordinator should have started. Can you see a “start topology coordinator fiber” log message on any of the nodes?
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** yes I see this on some of the nodes, but definitely not on all of them
> 
> ![Piotr_Dulikowski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/0/059db617d110231ced9a984559a6da500b22c526.png) **@Piotr\_Dulikowski:** It’s expected - the topology coordinator fiber runs on the current raft leader, so this won’t be printed on all nodes. So, most likely the upgrade process has started but got stuck.  
> I think the best way forward would be to do what @Kamil\_Braun suggested, so that we can analyze logs in more detail ourselves and look for more clues on what made it stuck.
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** looks like raft still sees 2 recently removed nodes:
> 
> ```auto
> [shard 0:strm] raft_topology - topology change coordinator fiber got error exceptions::unavailable_exception (Cannot achieve consistency level for cl ALL. Requires 28, alive 26)
> 
> ```
> 
> nodetool status gives 26. these 2 nodes were removed before upgrade to 6.0.3  
> how to convince raft that these 2 nodes are removed?  
> ahhh! After changing replication settings for system tables upgrade was successful. Thank you guys!
> 
> ![Piotr_Smaroń](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/c7e93f1f72e94abeeb7572b9785f007b82cc7308.png) **@Piotr\_Smaroń:** @rfurmanski what exactly have you changed, can you please share the details?
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:**
> 
> ```auto
> scyllamaster@cqlsh> alter keyspace system_auth WITH replication = {'class': 'NetworkTopologyStrategy','ams3': '6', 'ams4': '8', 'ash': '8','trn': '4'} AND durable_writes = 'true';
> scyllamaster@cqlsh> alter keyspace system_distributed WITH replication = {'class': 'NetworkTopologyStrategy
> ','ams3': '6', 'ams4': '8', 'ash': '8','trn': '4'} AND durable_writes = 'true';
> scyllamaster@cqlsh> alter keyspace system_traces WITH replication = {'class': 'NetworkTopologyStrategy','am
> s3': '6', 'ams4': '8', 'ash': '8','trn': '4'} AND durable_writes = 'true';
> 
> ```
> 
> previously in ams3 I had 8 nodes  
> after executing these commands raft upgrade went through
> 
> ![Kamil_Braun](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/16acd791289c6131034d6210e28a3368c9847a99.png) **@Kamil\_Braun:** ah, so it was probably trying to migrate from system\_auth to the new auth v2 tables  
> and it was trying to read system\_auth using CL=ALL
> 
> ![rfurmanski](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/3/3ed796fea230907083d50de24b0ee7c99c79ddc1.jpeg) **@rfurmanski:** yes
> 
> ![Kamil_Braun](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/16acd791289c6131034d6210e28a3368c9847a99.png) **@Kamil\_Braun:** we should probably improve that error message
> 
> ```auto
> [shard 0:strm] raft_topology - topology change coordinator fiber got error exceptions::unavailable_exception (Cannot achieve consistency level for cl ALL. Requires 28, alive 26)
> 
> ```
> 
> so it’s more clear which table it is trying to query