How do I change the node instance size in a running cluster?

ScyllaDB
,
1

I have a Scylla cluster running on AWS i3en.xlarge instances.

How can I change the instances to a different size, for example, i3en.2xlarge?
One way would be to replace each node, that is, to add and remove a node.
Is there a better way to do it?

*Based on a question originally asked on Stack Overflow by SilentCanon

2

Indeed, one way to do it would be to add a larger node, wait for the data to stream and then remove a small node. You can read more about this procedure in how to Upscale a Cluster. Another way to do this without having to change the nodes one by one would be to:

  1. Add a new DC to the existing cluster with the desired instance type. You can read more about this procedure here. It includes:

    • Install nodes on the new DC.
    • Add the new nodes one by one to the cluster.
    • Update the replication strategy of the selected keyspace/keyspaces to use with the new DC.
    • Rebuild new nodes
    • Run full cluster repair
    • Update the Monitoring stack

  2. Wait for the streaming to the new DC to finish

  3. Run a full cluster repair and wait for it to complete

  4. Decommission the “original” Datacenter by following the Remove a Data-Center from a Scylla Cluster procedure

Notice that it’s currently not recommended to use instances of different types in a single cluster. In the future, and by using tablet partitioning, this will change. We’re working on a new partitioner alongside Raft, which would reduce the total number of tokens or partitions, while still keeping the number high to guarantee even distribution of work. It will also allow balancing the data between partitions more flexibly. These partitions are called tablets. You can read more about this in this blog post.

*Based on an answer on Stack Overflow by TomerSan

3

Clarifying question: the linked “Adding a New Data Center Into an Existing ScyllaDB Cluster” doc requires using Ec2MultiRegionSnitch or the GossipingPropertyFileSnitch. Is that a hard requirement in the context of the answer here, where a new data center is being created to replace the old?

Ec2MultiRegionSnitch requires using public IP addresses, and I’m working with a cluster where public net access is disallowed for security. The EC2Snitch docs show it being used with multiple DC, so I believe it would work… but I wanted to confirm before disregarding that part of the instructions.

4

Since locator: Do not enforce public ip address for broadcast_rpc_address · scylladb/scylladb@77b1db4 · GitHub Ec2MultiRegionSnitch no longer enforces a public IP address.

You may also want to look into Ec2MultiRegionSnitch should not enforce a public RPC address · Issue #10236 · scylladb/scylladb · GitHub as it details that both Ec2Snitch and GossipingPropertyFileSnitch can still be used.

1 Like