This short report brings to light some interesting commits to scylladb.git master from the last week. Commits in the 885cb2af07b…c01b19fcb3f range are covered.
There were 66 non-merge commits from 18 authors in that period. Some notable commits:
Bloom filters are used to determine which sstables do not contain a partition key, speeding up reads when sstables can be filtered out. Since they are held in memory, and their size depends on the data (small partitions require larger bloom filters), their memory usage can overwhelm a node. The database will not track the aggregate memory consumption by bloom filters, and drop some filters if memory usage exceeds a fraction of the memory allocated to the shard.
Repair history is used to determine which tombstones can be garbage collected, as we only garbage collect tombstones that have been written before the last repair. We now load the repair history table in the background, so we don’t slow node start-up.
We now remember which auth tables version are in use, to avoid unnecessary version migration on startup.
In maintenance mode, we will skip loading tablet metadata if corrupted, to allow an administrator to fix it.
The large cell/row detector will no longer log partition keys as they can be sensitive; they are still written to the system.large_rows (and similar) tables, where access can be restricted.
Alternator, ScyllaDB’s implementation of the DynamoDB API, moved back from tablets to vnodes as its default replication method. This is because alternator requires LWT in some configurations, which is not yet supported with tablets.
Tablets now use a more compact data structure for the mapping between tablets and compaction groups; this helps conserve memory with extremely large tables.
An ALTER TABLE statement that increases the replication factor will now rebuild the table, if it uses tablets.
Tracking of memory usage during repair is now more accurate.
See you in the next issue of last week in scylladb.git master!