On the nodetool repair page there is a warning against doing basically anything to the cluster (maintenance or schema changes) during a repair: source
The applications that we run can dynamically change schemas, it’s rare after the initial start up (just creating their tables), but we might drop a table here or there and the apps will recreate it whenever they next run. This basically means that if I want to guarantee that there are no schema changes, I have to stop the world. This is impossible to do on the kind of schedule that repairs need to be run.
My question mainly is what kind of error would repair throw if a schema changed? Is this just something that it will kill the repair and we’d need to rerun it? Or could this cause other side effects in the cluster or data that we would need to recover from?
Adding or dropping tables while a repair is running should be fine. These tables may or may not be picked up by the currently running repair, depending on timing. But Scylla should be able to handle this case without problems.
What is dangerous is altering existing tables during a repair. Repair might still be using the old schema and it can choke on data written after the table alter, as it will possibly have columns, repair will not be able to process (due to the old schema).
Ok, just to clarify though, this will just result in the repair failing. Then a rerun of the repair should be fine (though will need to do the full set of data again). Correct?
Altering a table while repair is running can also result in a crash, or memory corruption in the worst case. This is quite unlikely but not at all impossible and we have seen it happening in the past.