The default SStable format used in version 5.1 is me
. However, 4.3, 4.4, 4.5, 4.6, and 5.0 default to md
. So when I upgrade all nodes in the cluster to 5.1, wrote some data, and finally rolled back to 4.6, the following error occurred:
ERROR 2023-09-11 07:12:46,448 [shard 0] database - Exception while populating keyspace 'alternator_ycsb' with column family 'ycsb' from file '/var/lib/scylla/data/alternator_ycsb/ycsb-1fa369f0506b11ee8364d7ed7565ea05': sstables::malformed_sstable_exception (invalid version for file me-4-big-Index.db. Name doesn't match any known version.)
ERROR 2023-09-11 07:12:46,451 [shard 0] database - Exception while populating keyspace 'system_distributed' with column family 'cdc_generation_timestamps' from file '/var/lib/scylla/data/system_distributed/cdc_generation_timestamps-fdf455c4cfec3e009719d7a45436c89d': sstables::malformed_sstable_exception (invalid version for file me-4-big-Index.db. Name doesn't match any known version.)
xxxxx
ERROR 2023-09-08 10:22:01,969 [shard 0] init - Startup failed: std::invalid_argument (Wrong sstable format name: me)
Although I have already recovered all tables in keyspace system
and system_schema
Finally, I tried to clean up all the me
format SStable. It still reports the following error:
2023-09-11T07:54:12.356630041Z ERROR 2023-09-11 07:54:12,356 [shard 0] init - Startup failed: std::runtime_error (Feature check failed. This node can not join the cluster because it does not understand the feature. Local node 100.71.8.124 features = {CDC, CDC_GENERATIONS_V2, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, CORRECT_STATIC_COMPACT_IN_MC, COUNTERS, DIGEST_FOR_NULL_VALUES, DIGEST_INSENSITIVE_TO_EXPIRY, DIGEST_MULTIPARTITION_READ, HINTED_HANDOFF_SEPARATE_CONNECTION, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, LWT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, MD_SSTABLE_FORMAT, NONFROZEN_UDTS, PER_TABLE_CACHING, PER_TABLE_PARTITIONERS, RANGE_SCAN_DATA_VARIANT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, UDA, UNBOUNDED_RANGE_TOMBSTONES, VIEW_VIRTUAL_COLUMNS, WRITE_FAILURE_REPLY, XXHASH}, Remote common_features = {AGGREGATE_STORAGE_OPTIONS, CDC, CDC_GENERATIONS_V2, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, CORRECT_STATIC_COMPACT_IN_MC, COUNTERS, DIGEST_FOR_NULL_VALUES, DIGEST_INSENSITIVE_TO_EXPIRY, DIGEST_MULTIPARTITION_READ, EMPTY_REPLICA_PAGES, HINTED_HANDOFF_SEPARATE_CONNECTION, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, LWT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, MD_SSTABLE_FORMAT, ME_SSTABLE_FORMAT, NONFROZEN_UDTS, PARALLELIZED_AGGREGATION, PER_TABLE_CACHING, PER_TABLE_PARTITIONERS, RANGE_SCAN_DATA_VARIANT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_COMMITLOG, SCHEMA_TABLES_V3, SEPARATE_PAGE_SIZE_AND_SAFETY_LIMIT, STREAM_WITH_RPC_STREAM, TOMBSTONE_GC_OPTIONS, TRUNCATION_TABLE, TYPED_ERRORS_IN_READ_RPC, UDA, UDA_NATIVE_PARALLELIZED_AGGREGATION, UNBOUNDED_RANGE_TOMBSTONES, VIEW_VIRTUAL_COLUMNS, WRITE_FAILURE_REPLY, XXHASH})
The official document provides a rollback method from 5.1 to 5.0. How does it avoid SStable formatting errors from me
to md
? What is our upgrade order? From 4.6 to 5.0 and then to 5.1? Then perform a reverse rollback.
Thakns!