Nodes reshaping data after compaction and not coming back online, reshaping progress indication

ScyllaDB
1

Hey all. Yesterday I forced a major compaction on our tables. Unfortunately, two of our nodes restarted and wouldn’t come back up online but spent their time reshaping data. Today we would like to get those nodes back online because we’re not seeing what the progress is. We tried running nodetool stop RESHAPE, but it seems like it just stops the reshaping task to then immediately start reshaping the same files again.

Is this potentially a bug? Any advice on how to get the nodes back online?

Running version 5.1 and SizeTieredCompactionStrategy.

*Originally asked on ScyllaDB’s community slack channel

2

It would be best if you could hold your breath and let reshaping finish.

3

Ok, cool, thanks. is there any way to get an indication (e.g., by looking at sizes on the filesystem) of the progress of the reshaping?

4

If you list the sstables on disk and cross-reference them with the log, you can see estimate how much progress reshape has made by looking at how many sstables were created before reshape started vs. those who were created after (and by reshape).

As for officially providing this information over the API, we’re working on a generic task manager that will provide an API to get tasks progress.

We’re currently working on integrating it with repair tasks, and next in line are node operations and compaction tasks.

5

Thanks! that’s what we ended up doing :slightly_smiling_face: happily, the reshape has finished by now.