# Raft majority loss issue - no raft quorum after node failure

**URL:** <https://forum.scylladb.com/t/raft-majority-loss-issue-no-raft-quorum-after-node-failure/2911>\
**Category:** ScyllaDB\
**Tags:** troubleshooting, administration, raft, multi-dc\
**Created:** [September 18, 2024, 4:28am UTC](https://forum.scylladb.com/t/raft-majority-loss-issue-no-raft-quorum-after-node-failure/2911 "2024-09-18T04:28:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\
**Post date:** [September 18, 2024, 4:28am UTC](https://forum.scylladb.com/t/raft-majority-loss-issue-no-raft-quorum-after-node-failure/2911/1 "2024-09-18T04:28:37Z")

</div>

> [@](#):
>
> [**Originally from the User Slack**](https://scylladb-users.slack.com/)
> 
> ![saranya_R_B](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/c/c0f005414c6ad64858116b14b35e8b4f7b5e6b30.png) **@saranya\_R\_B:** Hello scylla Team,  
> Scylla\_version: 6.0 keyspace enabled with tablets  
> We have configured two data centers with three nodes each, but one data center has completely failed and cannot be recovered. We’ve tried using `removenode` with the `--ignore-dead-nodes` option and the `replace_node_at_first_boot` procedure, but neither has been successful in removing or adding new nodes. Instead, we’re encountering the following error. Can anyone help with this?
> 
> "there is no raft quorum, total voters count 6, alive voters count 3, dead voters  
> **f-390e10ddc662] raft operation [read\_barrier] timed out, there is no raft quorum, total voters count 6, alive voters count 3, dead voters"**
> 
> ![Piotr_Smaroń](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/c7e93f1f72e94abeeb7572b9785f007b82cc7308.png) **@Piotr\_Smaroń:** cc @Kamil\_Braun
> 
> ![Botond_Dénes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/873f80169aadc07303ca5e860a0fca5146ab38c6.jpeg) **@Botond\_Dénes:** What do you mean exactly by “one data center has completely failed”? Did you loose all nodes in that DC?
> 
> # ![saranya_R_B](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/c/c0f005414c6ad64858116b14b35e8b4f7b5e6b30.png) **@saranya\_R\_B:** yes all nodes are down in one dc. we are running multi-dc architecture . Datacenter: SSD\_datacenter
> 
> # Status=Up/Down |/ State=Normal/Leaving/Joining/Moving – Address Load Tokens Owns Host ID Rack UN 10.92.3.5 94.17 GB 256 ? a86af8b2-7b56-44a9-80b6-c3fb1852ad01 rack\_ssd UN 10.92.3.6 93.28 GB 256 ? b4362683-532d-4185-a5b9-e2b8acb0658e rack\_ssd UN 10.92.3.7 93.92 GB 256 ? e1284d67-ca46-4e4b-a396-75fdd8142e61 rack\_ssd Datacenter: local\_ssd\_dc
> 
> Status=Up/Down  
> |/ State=Normal/Leaving/Joining/Moving  
> – Address Load Tokens Owns Host ID Rack  
> DN 10.92.3.8 81.08 GB 256 ? 7deb42d7-dd1d-4848-97d0-88462cf442ae local\_ssd\_rack  
> DN 10.92.3.9 82.17 GB 256 ? 26c1a8d3-e30f-496e-8509-d658ad758eda local\_ssd\_rack  
> DN 10.92.3.10 82.79 GB 256 ? fcde1ff7-05f4-44bd-b3d6-496f7a409ece local\_ssd\_rack
> 
> Using local ssd for reads and peristant disk for writes .ALll nodes in local\_ssd\_dc went down.  
> **Removal of node**  
> We’ve tried using `removenode` with the `--ignore-dead-nodes` option and the `replace_node_at_first_boot` procedure, but neither has been successful in removing or adding new nodes. Instead, we’re encountering error.
> 
> ![Botond_Dénes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/873f80169aadc07303ca5e860a0fca5146ab38c6.jpeg) **@Botond\_Dénes:** If you have two DCs, then you ran into the problem of raft majority loss. This is why it is advisable to always have odd number DCs, so you always have a majority left.
> 
> ![avi](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/b5e08becda8d9df1abff1d3a2d9ffe1126ac3fc6.jpeg) **@avi:** You need to run the raft recovery procedure to reassemble the cluster
> 
> ![Kamil_Braun](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/2X/1/16acd791289c6131034d6210e28a3368c9847a99.png) **@Kamil\_Braun:** [https://opensource.docs.scylladb.com/branch-6.0/troubleshooting/handling-node-failures.html#manual-recovery-procedure](https://opensource.docs.scylladb.com/branch-6.0/troubleshooting/handling-node-failures.html#manual-recovery-procedure)
> 
> > Handling Node Failures | ScyllaDB Docs
> 
> ah actually it won’t work with tablets  
> \> The manual recovery procedure is not supported [if tablets are enabled on any of your keyspaces](https://opensource.docs.scylladb.com/branch-6.0/architecture/tablets.html). In such a case, you need to [restore from backup](https://opensource.docs.scylladb.com/branch-6.0/operating-scylla/procedures/backup-restore/restore.html).
> 
> > Data Distribution with Tablets | ScyllaDB Docs
> 
> > Restore from a Backup and Incremental Backup | ScyllaDB Docs
> 
> in this case the only way is to setup a new cluster and move the data over, unfortunately