# Tombstones, gc\_grace\_seconds, propagation\_delay\_in\_seconds, TWCS, compaction and related questions **URL:** \ **Category:** ScyllaDB\ **Tags:** data-model, compaction, twcs, tombstone\ **Created:** [December 4, 2024, 4:32am UTC](https://forum.scylladb.com/t/tombstones-gc-grace-seconds-propagation-delay-in-seconds-twcs-compaction-and-related-questions/4064 "2024-12-04T04:32:19Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![Guy](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/guy/32/9_2.png) [@Guy](https://forum.scylladb.com/u/Guy)\ **Post date:** [December 4, 2024, 4:32am UTC](https://forum.scylladb.com/t/tombstones-gc-grace-seconds-propagation-delay-in-seconds-twcs-compaction-and-related-questions/4064/1 "2024-12-04T04:32:19Z")
> [@](#): > > [**Originally from the User Slack**](https://scylladb-users.slack.com/) > > ![Artem_Golovko](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/657780876ac3a9aa8c606bb8004e7a27199a1fd7.png) **@Artem_Golovko:** Hello everyone, > > I’m so confused in understanding how tombstones works in scylla and will be really happy if someone could reveal my doubts. > > 1. I’m using **scylla version 6.1.2** > **1. Tombstone “validity”** > As far as I understood to prevent data resurrection scylla uses some technics to prevent tombstones to be removed (tombstone\_gc). > a) timeout. Uses “gc\_grace\_seconds” table option > **Q1** : Does gc\_grace\_seconds is using only for tombstone\_gc = timeout? > **Q2** : Does gc\_grace\_seconds is using for something else? > b) repair. Tombstone can be removed only if repair was done. I found the following in the documentation “For tables which use tablets repair mode is set by default.”. Tablets now > are default, so I expected this will be a default > **Q3** : but when I described the table I can see “AND tombstone\_gc = {‘mode’: ‘timeout’, ‘propagation\_delay\_in\_seconds’: ‘3600’}”. Why mode = timeout? > **Q4** : what is propagation\_delay\_in\_seconds? I can’t find any documentation about it > > c) disabled - not interested > > d) immediate > **Q5** : why it’s safe to use for TWCS? > **Q6** : how it’s differ from the mode = timeout and gc\_grace\_seconds = 0? > > **2. When tombstone and underlying data can be removed?** > a) During tables compaction > **Q7** : tombstone and underlying data can be removed if they are compacted together to a new SSTable, right? > > b) During tombstone compaction when only one SSTable itself got compacted to remove data. > **Q8** : Does scylla has tombstone compaction? If so, what the strategy or in other words when this occurs? > **Q9** : Does scylla supports nodetool garbagecollect? > **Q10** : From TWCS documentation “Tombstone compaction can be enabled to remove data from partially expired SSTables, but this creates additional WA (write amplification).”. How it can be enabled? > **Q11** : Does tombstone compaction enabled with tombstone\_threshold, tombstone\_compaction\_interval and unchecked\_tombstone\_compaction options? Also, would like to understand more about these options. > **Q12** : if I’m going to delete data with CL=ALL will tombstone created? And how to avoid tombstone creation and force scylla delete data immediately? > **Q13** : Using TWCS how to delete data immediately to avoid tombstones? I have a use case when in rare cases I need to remove entire data by partition key or by partition key and clustering range. So, you’ll have tombstone in one windowed SSTable, but actual data in another windowed SSTable. According to the strategy SSTables from different windows never compacted. It means I need tombstone compaction. Currently it leads to the bad performance and it looks like scylla does not have “tombstone compaction” at all or I did something wrong, > > 1. I found that in the documentation everything refer to the gc\_grace\_seconds, but there is no documentation about when tombstone compaction will occur if will. > > 1. [https://opensource.docs.scylladb.com/stable/kb/ttl-facts.html](https://opensource.docs.scylladb.com/stable/kb/ttl-facts.html) > 2. recent discord video about “Application tombstone” ([https://www.youtube.com/watch?v=60GnK5sHgDY](https://www.youtube.com/watch?v=60GnK5sHgDY)) also use gc\_grace\_seconds = 0, that’s kind of strange, because it’s does not provide control when actually data will be removed and also on that time “repair” mode already release. Why they did not use it? > Also created a github issue for that [https://github.com/scylladb/scylladb/issues/21541](https://github.com/scylladb/scylladb/issues/21541) > > ![Felipe_Cardeneti_Mendes](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/233a9d524fbc97e39f08633c16b3dd2575d1d1f4.jpeg) **@Felipe\_Cardeneti\_Mendes:** **Q1** : Does gc\_grace\_seconds is using only for tombstone\_gc = timeout? > Yes > **Q2** : Does gc\_grace\_seconds is using for something else? > No. > > **Q3** : but when I described the table I can see “AND tombstone\_gc = {‘mode’: ‘timeout’, ‘propagation\_delay\_in\_seconds’: ‘3600’}”. Why mode = timeout? > Might be an already fixed bug, it should be `repair` in this case. Did you enabled tablets in the corresponding keyspace? Either way, 6.2 should have it already changed, otherwise fill an issue. > > **Q4** : what is propagation\_delay\_in\_seconds? I can’t find any documentation about it > How long after repair completes compaction is free to evict tombstones. We can’t immediately evict them, due to out of order writes. > > **Q5** : why it’s safe to use for TWCS? > TWCS assumes no data deletes and append-only. Thus, as soon as TTL expires compaction is free to purge tombstones. If that’s not the case, you shouldn’t be using TWCS in the first place. > > **Q6** : how it’s differ from the mode = timeout and gc\_grace\_seconds = 0? > There’s a good thelastpickle article which touches on some of the problems involving gc\_grace=0, in particular related to hints replay. > > **Q7** : tombstone and underlying data can be removed if they are compacted together to a new SSTable, right? > Right. > > **Q8** : Does scylla has tombstone compaction? If so, what the strategy or in other words when this occurs? > We do. See [https://opensource.docs.scylladb.com/stable/cql/compaction.html#common-options](https://opensource.docs.scylladb.com/stable/cql/compaction.html#common-options) for the rest of the answer. > > **Q9** : Does scylla supports nodetool garbagecollect? > No. You can run a major, tho. > > **Q10** : From TWCS documentation “Tombstone compaction can be enabled to remove data from partially expired SSTables, but this creates additional WA (write amplification).”. How it can be enabled? > See link from Q8 > > **Q11** : Does tombstone compaction enabled with tombstone\_threshold, tombstone\_compaction\_interval and unchecked\_tombstone\_compaction options? Also, would like to understand more about these options. > The logic is: You configure how often you want compaction to check for SSTable eligible for tombstone compaction, you set a ratio for the single SSTable to be garbage-collected. unchecked\_tombstone\_compaction disables it altogether. > > **Q12** : if I’m going to delete data with CL=ALL will tombstone created? And how to avoid tombstone creation and force scylla delete data immediately? > Tombstone is created irrespective of your CL. Use immediate mode and they will be evicted on flush if you use CL=ALL. > > **Q13** : Using TWCS how to delete data immediately to avoid tombstones? I have a use case when in rare cases I need to remove entire data by partition key or by partition key and clustering range. So, you’ll have tombstone in one windowed SSTable, but actual data in another windowed SSTable. According to the strategy SSTables from different windows never compacted. It means I need tombstone compaction. Currently it leads to the bad performance and it looks like scylla does not have “tombstone compaction” at all or I did something wrong, > > You applied a tombstone to a different compaction window when you deleted. Compaction windows are never compacted together. You must run a major and re-asses the need for TWCS. > > ![Artem_Golovko](https://us1.discourse-cdn.com/flex016/uploads/scylladb/original/1X/657780876ac3a9aa8c606bb8004e7a27199a1fd7.png) **@Artem_Golovko:** @Felipe\_Cardeneti\_Mendes so many thanks for the answers!