The tombston_gc={'mode': 'repair'} option doesn't remove tombstones

ScyllaDB
, ,
1

I’m using ScyllaDB 5.4.x version.
I tested the tombstone_gc={‘mode’: ‘repair’} option and found it doesn’t remove tombstones.

I had three ScyllaDB nodes and created keyspace and table like this.

CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy', 'datacenter1': '3'}  AND durable_writes = true;

CREATE TABLE test.gc_test (
    id int PRIMARY KEY,
    msg text
) WITH tombstone_gc = {'mode': 'repair'}
    AND compaction = {'class': 'SizeTieredCompactionStrategy'}
    AND default_time_to_live = 0
    AND gc_grace_seconds = 0;

And insert and delete datas to gc_test table.

INSERT INTO gc_test (id, msg) VALUES (1, 'hello world');
INSERT INTO gc_test (id, msg) VALUES (2, 'bye world');
DELETE FROM gc_test WHERE id = 2;

Then I executed nodetool flush, so I could see the tombstone from sstable.

{
  "sstables": {
    "data/test/gc_test-d7ce27b0f62c11ee95e74a9b39589d16/me-3gf5_0dm4_1piia2a2dpi5kl13iu-big-Data.db": [
      {
        "key": {
          "token": "-4069959284402364209",
          "raw": "000400000001",
          "value": "1"
        },
        "clustering_elements": [
          {
            "type": "clustering-row",
            "key": {
              "raw": "",
              "value": ""
            },
            "marker": {
              "timestamp": 1712638319527618
            },
            "columns": {
              "msg": {
                "is_live": true,
                "type": "regular",
                "timestamp": 1712638319527618,
                "value": "hello world"
              }
            }
          }
        ]
      },
      {
        "key": {
          "token": "-3248873570005575792",
          "raw": "000400000002",
          "value": "2"
        },
        "tombstone": {
          "timestamp": 1712638377934685,
          "deletion_time": "2024-04-09 04:52:57z"
        }
      }
    ]
  }
}

Then I executed nodetool repair to all nodes and expected the tombstone to be removed, but it was still there.

Why hasn’t the tombstone been removed?

2

You need a compaction after the repair has happened, for the tombstone to be dropped. You can force a major compaction here (nodetool compact) to be able to observe this.

3

I forced major compaction, but the tombstone still remained.

4

Tombstones are eligible for being GC-ed if they are older than repair start timestamp - 1h. So they will disappear if you run repair 1h after deletion happened and then you compact the sstables.

5

This 1h grace period is in order to account for the fact that there could be writes covered by tombstones, which are “stuck” in various pipelines during repair, and we don’t want to drop tombstones before those writes reach replicas and are visible to compaction. Otherwise, those writes, which should be dropped, would be resurrected.

1 Like