Hello everyone,
I’m so confused in understanding how tombstones works in scylla and will be really happy if someone could reveal my doubts.
- I’m using scylla version 6.1.2
- Tombstone “validity”
As far as I understood to prevent data resurrection scylla uses some techniques to prevent tombstones to be removed (tombstone_gc).
a) timeout. Uses “gc_grace_seconds” table option
Q1: Does gc_grace_seconds is using only for tombstone_gc = timeout?
Q2: Does gc_grace_seconds is using for something else?
b) repair. Tombstone can be removed only if repair was done. In another word Scylla remembering last repair time and only tombstones that are older than last repair can be removed. I found the following in the documentation “For tables which use tablets repair mode is set by default.”. Tablets now are default, so I expected this will be a default mode
Q3: but when I described the table I can see “AND tombstone_gc = {‘mode’: ‘timeout’, ‘propagation_delay_in_seconds’: ‘3600’}”. Why mode = timeout?
Q4: what is propagation_delay_in_seconds? I can’t find any documentation about it
c) disabled - not interested
d) immediate
Q5: why it’s safe to use for TWCS?
Q6: how it’s differ from the mode = timeout and gc_grace_seconds = 0? - When tombstone and underlying data can be removed? In another words: in what process tombstones and underlying data are removing.
a) During tables compaction
Q7: tombstone and underlying data can be removed if they are compacted together to a new SSTable, right?
b) During tombstone compaction when only one SSTable itself got compacted to remove data.
Q8: Does scylla has tombstone compaction? If so, what the strategy or in other words when this occurs?
Q9: Does scylla supports nodetool garbagecollect?
Q10: From TWCS documentation “Tombstone compaction can be enabled to remove data from partially expired SSTables, but this creates additional WA (write amplification).”. How it can be enabled?
Q11: Does tombstone compaction enabled with tombstone_threshold, tombstone_compaction_interval and unchecked_tombstone_compaction options? Also, would like to understand more about these options.
Q12: if I’m going to delete data with CL=ALL will tombstone created? And how to avoid tombstone creation and force scylla delete data immediately?
Q13: Using TWCS how to delete data immediately to avoid tombstones? I have a use case when in rare cases I need to remove entire data by partition key or by partition key and clustering range. So, you’ll have tombstone in one windowed SSTable, but actual data in another windowed SSTable. According to the strategy SSTables from different windows never compacted. It means I need tombstone compaction. Currently it leads to the bad performance and it looks like scylla does not have “tombstone compaction” at all or I did something wrong, - I found that in the documentation everything refer to the gc_grace_seconds, but there is no documentation about when tombstone compaction will occur if will.
- opensource .docs.scylladb.com/stable/kb/ttl-facts.html
- recent discord video about “Application tombstone” (youtube .com/watch?v=60GnK5sHgDY) also use gc_grace_seconds = 0, that’s kind of strange, because it’s does not provide control when actually data will be removed and also on that time “repair” mode already released (and maybe in their version already a default). Why they did not use it and relying on the “gc_grace_seconds” option?