Safe way change table compaction strategy

ScyllaDB
, ,
1

Installation details
#ScyllaDB version: 5.1.19
#Cluster size: 22
#OS: ubuntu

Hello everyone,

My Scylla cluster version is 5.1. Due to some performance issues,
I want to change the compaction strategy of certain tables from LCS (Leveled Compaction Strategy) to STCS (Size-Tiered Compaction Strategy).
In my tests, Scylla triggers a major compaction to apply the new compaction strategy,
but I want to minimize the impact during this transition.
Additionally, I would like a rollback mechanism in case of any issues with Scylla. Here’s the approach I’m considering:

  1. Disable auto-compaction on all nodes.
  2. Alter the table compaction strategy from LCS to STCS.
  3. Enable auto-compaction on one node. After confirming that all pending tasks are complete (conversion done), enable auto-compaction on the next node.

If any issues occur, I can revert the table compaction strategy back to LCS.

image
image587×437 21.5 KB

Here are my questions:

Q1. Is this process okay, or is there a better approach?

Q2. If I disable auto-compaction for an extended period (possibly up to 10 days, due to the large size of my data), could this cause any issues?

2

First off, please update to a more recent version, 5.1 is EOL for a long time now.

I think you are overly cautious. Yes, changing the compaction strategy will trigger a major compaction (a reshape actually), but ScyllaDB’s scheduling groups isolation should mean that this doesn’t have adverse affects on your cluster. If you want to be extra sure, you can set compaction_static_shares: 100 in the config, to make sure compaction doesn’t become too aggressive.

Note that disabling compaction, or throttling it down to much has its own risks: sstables can start accumulating and reads will start to have higher latencies and take more memory. This is especially true when switching compaction strategies and the sstables are out-of-shape compared to what the new compaction strategy expects. So I don’t recommend disabling auto compaction for any extended period of time.