Why flush_hints when gc_mode is repair?

ScyllaDB
,
1

I saw in repair.cc that flush_hints are required when tombstone_gc_mode is repair.
When writing fails, request data will be recorded in the hints directory. Does flush_hints flush these records to disk?

2

Flushing hints and batchlog when repairing with tombstone_gc is required to prevent any data in either causing data resurrection. With tombstone_gc, any tombstone that was written before the last repair, can be garbage-collected, so it is important to include any data that is in the batchlog or in hints in the repair, so any tombstone that might shadow them can take effect.

1 Like
3

Specifically, to ensure that no data resurrection occurs in which scenario? I found that not performing the flush_hints operation has no effect, as shown in the following figure. Because in order to clear the tombstone, all replicas must participate in the repair. So even without flush_hints, data that fails to be written will still be repaired, including tombstones.

image
image795×317 6.59 KB

1 Like
4

Hints can contain data which is deleted by a tombstone. Repair runs, the tombstone is GC’d, then hints are replayed and data is resurrected. This is why hints have to be replayed before repair.

1 Like
5

If coordinator (A) fails to write a=1 to node B, then A records the hints. After a period of time, there is a request to delete a=1 and write it. If the deletion fails, hints will also be remembered. And then the unified flush_hints have no effect. If the write is successful, it means B is alive, and the hints will also be sent over. If the write is successful, it means B is alive, and the hints will also be sent over. This won’t have any impact, right? So why do we have to flushvints?

6

Flushing hints is a safety mechanism, in case some hints linger on the nodes. Normally, there wouldn’t be any, this is to cover corner cases.

7

I can’t figure it out no matter what. Can you give me a simple example? Thank you.
In other words, what are the consequences of not executing flush_hints? How is data resurrected?

8

Taking your example from:

Node (A) may miss the delete a. It may not notice (B) becoming online. Sending the hints may fail. These are unlikely but it is very hard to prove they cannot happen. So we usually program defensively and want to make sure there are no hints left on any node after a repair, so data resurrection is off the table.