# The tombston\_gc={'mode': 'repair'} option doesn't remove tombstones

**URL:** <https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471>\
**Category:** ScyllaDB\
**Tags:** open-source, nodetool, tombstone\
**Created:** [April 9, 2024, 5:57am UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471 "2024-04-09T05:57:36Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sonny](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/sonny/32/545_2.png) [@sonny](https://forum.scylladb.com/u/sonny)\
**Post date:** [April 9, 2024, 5:57am UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471/1 "2024-04-09T05:57:36Z")

</div>

I’m using ScyllaDB 5.4.x version.  
I tested the tombstone\_gc={‘mode’: ‘repair’} option and found it doesn’t remove tombstones.

I had three ScyllaDB nodes and created keyspace and table like this.

```auto
CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'} AND durable_writes = true;

CREATE TABLE test.gc_test (
    id int PRIMARY KEY,
    msg text
) WITH tombstone_gc = {'mode': 'repair'}
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0;

```

And insert and delete datas to gc\_test table.

```auto
INSERT INTO gc_test (id, msg) VALUES (1, 'hello world');
INSERT INTO gc_test (id, msg) VALUES (2, 'bye world');
DELETE FROM gc_test WHERE id = 2;

```

Then I executed `nodetool flush`, so I could see the tombstone from sstable.

```auto
{
  "sstables": {
    "data/test/gc_test-d7ce27b0f62c11ee95e74a9b39589d16/me-3gf5_0dm4_1piia2a2dpi5kl13iu-big-Data.db": [
      {
        "key": {
          "token": "-4069959284402364209",
          "raw": "000400000001",
          "value": "1"
        },
        "clustering_elements": [
          {
            "type": "clustering-row",
            "key": {
              "raw": "",
              "value": ""
            },
            "marker": {
              "timestamp": 1712638319527618
            },
            "columns": {
              "msg": {
                "is_live": true,
                "type": "regular",
                "timestamp": 1712638319527618,
                "value": "hello world"
              }
            }
          }
        ]
      },
      {
        "key": {
          "token": "-3248873570005575792",
          "raw": "000400000002",
          "value": "2"
        },
        "tombstone": {
          "timestamp": 1712638377934685,
          "deletion_time": "2024-04-09 04:52:57z"
        }
      }
    ]
  }
}

```

Then I executed `nodetool repair` to all nodes and expected the tombstone to be removed, but it was still there.

Why hasn’t the tombstone been removed?

---

<div class="post-metadata">

**Author:** ![Botond\_Denes](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/botond_denes/32/163_2.png) [@Botond\_Denes](https://forum.scylladb.com/u/Botond_Denes)\
**Post date:** [April 15, 2024, 9:02am UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471/2 "2024-04-15T09:02:15Z")

</div>

You need a compaction after the repair has happened, for the tombstone to be dropped. You can force a major compaction here (`nodetool compact`) to be able to observe this.

---

<div class="post-metadata">

**Author:** ![sonny](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/sonny/32/545_2.png) [@sonny](https://forum.scylladb.com/u/sonny)\
**Post date:** [April 17, 2024, 7:27am UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471/3 "2024-04-17T07:27:04Z")

</div>

I forced major compaction, but the tombstone still remained.

---

<div class="post-metadata">

**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\
**Post date:** [April 22, 2024, 10:31pm UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471/4 "2024-04-22T22:31:54Z")

</div>

Tombstones are eligible for being GC-ed if they are older than repair start timestamp - 1h. So they will disappear if you run repair 1h after deletion happened and then you compact the sstables.

---

<div class="post-metadata">

**Author:** ![tgrabiec](https://sea2.discourse-cdn.com/flex016/user_avatar/forum.scylladb.com/tgrabiec/32/61_2.png) [@tgrabiec](https://forum.scylladb.com/u/tgrabiec)\
**Post date:** [April 22, 2024, 10:37pm UTC](https://forum.scylladb.com/t/the-tombston-gc-mode-repair-option-doesnt-remove-tombstones/1471/5 "2024-04-22T22:37:02Z")

</div>

This 1h grace period is in order to account for the fact that there could be writes covered by tombstones, which are “stuck” in various pipelines during repair, and we don’t want to drop tombstones before those writes reach replicas and are visible to compaction. Otherwise, those writes, which should be dropped, would be resurrected.