The question is about the nature of a row invalidation process in the row cache.
We have the corresponding monitoring metric - scylla_cache_row_removals, and I’d like to understand what this number should mean for me.
The basic questions are:
What is the list of events leading to the row invalidation in the row cache?
Does the row eviction always follow the row invalidation (with the corresponding scylla_cache_row_evictions incrementation)?
Is it possible to have row eviction without the row invalidation (probably yes, but just in case)?
scylla_cache_row_removals is incremented whenever a row is removed from the cache for reasons other than eviction. That may be a bit confusing, since “eviction” also removes the row from the cache, but it is not counted in scylla_cache_row_removals.
scylla_cache_row_evictions is typically incremented when a row entry is removed due to memory pressure to free space for new memory allocations.
It is also incremented when cache reader decides that a so called “dummy” row entry is redundant during scan. Those row entries hold no data but serve as anchors which represent boundaries of the clustering range used by the read. When a scan for a given clustering key range populates the cache, we insert dummy entries for the boundaries of the range in order to store the information that the whole range is continuous (populated).
Because we have those “dummy” rows, which are included in the metrics, the number of row evictions or removal may be greater than what you would expect from the number of CQL rows. Each partition has at least 1 dummy row entry (last dummy). In particular, a single-row partition has 2 row entries. The last dummy is special, because it is never evicted, always removed with the partition version.
scylla_cache_row_removals is incremented when a row entry is removed due to following reasons:
partition holding the row was “invalidated”, typically because the sstable set was modified for reasons other than memtable flush: repair, sstable upload. Data in cache may no longer reflect the state of sstables, so we remove affected partitions.
table was dropped
after memtable flush, when we apply data from memtable to cache, and the row was covered by a tombstone or overwritten.