The test found that when expanding two nodes at the same time into the cluster, one node will report an error. Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while consistent_rangemovement is true (check_for_endpoint_collision).
What will happen if we change consistent_rangemovement to false? Support two nodes bootstrap at the same time, what will be the bad effects? Is this a production-ready solution?
What will happen if we
change consistent_rangemovementtofalse? Support two nodes bootstrap at the same time, what will be the bad effects? Is this a production-ready solution?
You shouldn’t turn off this option, as it means you will disable consistent range movements (or enable inconsistent range movements ;-))
Consider you have nodes N1, N2, N3. You bootstrap N4 & N5, consider N4 started first and you started N5 before N4 finished.
N4 will stream data from N1~N3 for the ranges it now owns. N5 bootstraps, places its vNodes into the ring similarly as N4 did. It will stream from N1~N4, but there may be ranges N4 hasn’t streamed yet.
Worse, what if N4 fails?
The general solution for these are Raft and Tablets.
You may read ScyllaDB’s Path to Strong Consistency: A New Milestone - ScyllaDB - which includes a nice Q&A in the end which addresses your question.
And for tablets: Why ScyllaDB is Moving to a New Replication Algorithm: Tablets - ScyllaDB
Also related: https://github.com/apache/cassandra/blob/cassandra-3.9/doc/source/operating/topo_changes.rst#range-streaming