Originally from the User Slack
@Shobhana: I have a 3-node cluster (on Ubuntu 24.04) running 6.2.2-0.20241128.c6ef055e9c3b with RF of 2 and a total data size of around 2TB (total disk space usage from all 3 nodes). I use global secondary indexes in my tables and I have queries which read using these indexes.
Recently, one of the nodes encountered an issue and was not working now. I added a new node and decommissioned the faulty node. After that, during my regular weekly maintenance window, I ran nodetool repair -pr on the first node which was successful. But on the 2nd node, it got stuck and never completed. CPU usage was also very high on the 2nd node. I tried to stop the repair task, but it did not work. I tried to restart the Scylla service, but that didn’t work either.
Since this is a very old version, I know I will not get any support. So I thought it would be easier to set up a new cluster with the latest version and copy the data from old cluster to the new cluster. I setup the new cluster (version 2025.4.3-0.20260205.92d2bc4b47ae) and used the Scylla DB Spark migration tool (Getting Started ) to copy the data from required keyspaces and tables.
To make the data copy process quicker, when I created the tables, I set the RF to 1 and did not create any global secondary indexes. Now after the data copy is complete, I am trying to add global secondary indexes, but it gives an error
InvalidRequest: Error from server: code=2200 [Invalid query] message=“Materialized views and secondary indexes are not supported on base tables with tablets. To be able to use them, enable the configuration option rf_rack_valid_keyspaces and make sure that the cluster feature VIEWS_WITH_TABLETS is enabled.”
What is the right way to fix it? I have old data which I cannot afford to loose.
Getting Started | ScyllaDB Docs
Can anyone please help and suggest the best way to resolve this? I cannot afford to loose the existing data. My queries use these global secondary indexes. How can I create these indexes and let my app server query based on the secondary indexes?
I followed these steps to copy the data from my old cluster to the new cluster:
a) Created keyspaces and tables in the new cluster with RF as 1. Did not create any global secondary indexes or materialized views
b) Used ScyllaDB Spark migrator to copy data from required keyspaces and tables
c) Now I am trying to create global secondary index, but it fails
@Piotr_DulikowskiPiotr_Dulikowski**:** In keyspaces that use tablets (our new data distribution mechanism, alternative to vnodes), materialized views and secondary indexes only work in a setup where you have one replica per rack. So, if you want to have RF=N, you need to set up a cluster with exactly N racks, enable the rf_rack_valid_keyspaces option on each node and then perform migration. This restriction will be relaxed a bit in the next version where you can have RF that is smaller than the number of racks, but in general we will allow at most one replica of a keyspace per rack.
In your case - considering that your cluster had three nodes but RF=2 - I think the simplest way out would be to re-create the keyspace without tablets and redo the migration process. The restriction that I mentioned is not relevant for vnodes - you can check that it is allowed by trying to create the secondary index right after creating the (empty) base table. Alternatively, you can consider changing your cluster’s topology to fit the rf_rack_valid_keyspaces restriction.
To create a keyspace which uses vnodes you need to do:
CREATE KEYSPACE <ks> WITH tablets = {'enabled': false};
Unfortunately, it is not possible to ALTER an existing keyspace to change from tablets to vnodes.
See https://docs.scylladb.com/manual/stable/architecture/tablets.html#enabling-tablets for more details.
Data Distribution with Tablets | Scyll@Piotr_DulikowskiDB Docs
@Shobhana: @Piotr_Dulikowski Thanks for taking time to look into my issue. I will create the keyspace again. What is the most efficient way to copy the data either from the old cluster to the new cluster? I had previously used ScyllDB Spark migrator when I migrated from my Dynamo DB to ScyllaDB and I used the same approach to copy data from old cluster to new cluster. Is there a better way considering the clusters are using different versions?
b) Is there an efficient way to copy the data from the old keyspace to the new keyspace within the same cluster? Can I create a new keyspace with a different name, copy the data from old keyspace to the new keyspace, delete the old keyspace and rename new keyspace to original name? If this can be done, I would prefer this to avoid adding burden on my old cluster which is serving production traffic now.
@Piotr_Dulikowski**:** According to our documentation, our supported options for migration are either to use the spark migrator or use the load and stream functionality: https://docs.scylladb.com/manual/stable/using-scylla/mig-tool-review.html. Load and stream should be more efficient than the spark migrator, if you can use it.
Simply copying the sstables from the old keyspace to the new keyspace will not work because vnodes will distribute the data between nodes a bit differently than tablets. I think you should be able to use load and stream with the sstables from the old keyspace, but I’m not an expert on this feature so I’m pinging @Botond_Dénes so that he can confirm or deny.
ScyllaDB Migration Tools: An Overview | ScyllaDB Docs
@Shobhana: Okay thanks; I’ll wait for suggestion from @Botond_Dénes
@Botond_Dénes: Yes, load-and-stream should work.
@Shobhana: Is there no way to use the option (b) mentioned in my previous reply?
@Botond_Dénes: Option (b) above is similar to in-place migration from vnodes to tablets, this is still WIP and no releases has it.
Current releases don’t have the logic to re-shape sstables from vnodes to tablets. So the only option is to use nodetool refresh --load-and-stream or something like the spark migrator or equivalent.
@Shobhana: Okay thanks. I’ll recreate the keyspaces by disabling the tablets and copy the data again from the old cluster