Unexpected Behavior with TimeWindowCompactionStrategy in ScyllaDB 6.2 Open Source

ScyllaDB
, , ,
1

Hi Community,

I’m using ScyllaDB 6.1 Open Source and have a table configured to store 30 days of data with the following compaction strategy:

compaction = {
** ‘class’: ‘TimeWindowCompactionStrategy’,**
** ‘compaction_window_size’: ‘3’,**
** ‘compaction_window_unit’: ‘DAYS’**
}

Currently, no reads are performed on this table—only data inserts.

Here’s what I observed:

  • Before today (January 7), the table had SSTables grouped in 3-day windows, such as (Dec 23, Dec 25, Dec 28, Dec 31, Jan 3).
  • Today (Jan 7), after triggering an autocompaction, I noticed that the compaction created a new table containing only data for Jan 7. The previous 3-day window grouping behavior seems to have changed unexpectedly.

I haven’t manually triggered any compaction apart from observing this autocompaction behavior.

Does anyone have insights into why this might have happened? Could it be related to some internal behavior or a specific setting I might have missed?

Any help or suggestions would be greatly appreciated!

2

I posted this question a few days ago, but I haven’t received a response yet. I wanted to provide some additional details about the issue I’m facing.

To recap:

I’m using ScyllaDB 6.1 Open Source and have a table configured to store 30 days of data with the following compaction strategy:

compaction = {
** ‘class’: ‘TimeWindowCompactionStrategy’,**
** ‘compaction_window_size’: ‘3’,**
** ‘compaction_window_unit’: ‘DAYS’,**
** ‘max_threshold’: ‘32’,**
** ‘min_threshold’: ‘4’**
}

Observations

  1. Pre-7 Jan Behavior: The table had SSTables grouped in 3-day windows, such as (Dec 23, Dec 25, Dec 28, Dec 31, Jan 3).
  2. On 7 Jan: After an autocompaction, a new SSTable was created for Jan 7 only, deviating from the expected 3-day grouping behavior.

Additional Issue

Upon further investigation, I noticed that instead of forming a single large SSTable for each 3-day window, there are multiple small SSTables within each window. These smaller SSTables are not being compacted into a single SSTable as I expected, even though the compaction strategy specifies min_threshold = 4 and max_threshold = 32.

Questions

  1. Why are these smaller SSTables within the same 3-day window not being compacted into one large SSTable?
  2. Are there specific conditions under which TimeWindowCompactionStrategy avoids compacting SSTables, even when the min_threshold and max_threshold values should allow it?
  3. Could there be an issue with autocompaction triggering or the way TimeWindowCompactionStrategy handles compaction windows for insert-only workloads?

I’d appreciate any insights or suggestions to troubleshoot and resolve this issue.

Thank you in advance!