Originally from the User Slack
@Samuel_Hameau: Hi, i’m having sstable expiration issues, i am forced to run manually compactions to gain space.
I’m having timeseries workload, with data inserted only, using ttl (of 1 week max), with compaction = TWCS/1/DAY and gc_grace_seconds = 36000.
Here is a graph showing data disk usage increase during 13d, where we can see automatic compaction during 10d (minor impact) and then daily manual compaction the last 3 days (stabilisation of disk usage).
I expected to get disk usage stagnation at some point, but i only manage to get it with manual compaction.
Is it anything I can do to optimize compaction/disk usage without running manual compaction ? (scylla 5.2.17 cluster with 2DC / 8 nodes, RF=3)
@avi: Is your workload TWCS compatible? Clustering keys are current time, no overwrites or deletes?
@Samuel_Hameau: I believe that yes: we are inserting data on the last 7 days regarding startdate (startdate can be between 7d until 5min before current time; and ajusting ttl accordingly to make it expire 7 days after startdate)
PRIMARY KEY is ((userid, devicetype, brand, yearmonth), startdate, deviceid, attrib)
There is no delete, there should probably never have any overwrites (but if it occurs, it would overwrite with same expiration date (to make it expires 7d after startdate)
@avi: Strange, maybe @raphaelsc can give debugging tips
@Samuel_Hameau: using sstabledump $sstable|grep expired.*false, i see that data is stored sorted regarding to insertion date, not regarding expiration:
"liveness_info" : { "tstamp" : "2024-09-22T02:56:36.355988Z", "ttl" : 888475, "expires_at" : "2024-10-02T09:44:31Z", "expired" : false },
"liveness_info" : { "tstamp" : "2024-09-22T06:54:13.314949Z", "ttl" : 1191787, "expires_at" : "2024-10-06T01:57:20Z", "expired" : false },
"liveness_info" : { "tstamp" : "2024-09-22T06:54:13.314949Z", "ttl" : 1180910, "expires_at" : "2024-10-05T22:56:03Z", "expired" : false },
As ttl varies at insertion date, that may explain why sstables are not removed (until one of the records is not expired) ?
@raphaelsc: 1180910 = ~13d. yes, twcs waits until sstable is fully expired, so when you mix ttl of different values, the data in a window will only be removed when all data expires. that’s the default behavior, you can force early purge, but with write amplification cost, by setting tombstone_threshold setting to 0.5 in the compaction options. that will allow twcs to remove data incrementally from a window.
@Samuel_Hameau: thanks for the hint about huge ttl, and for the tombstone_threshold suggestion 