The default SStable format used in version 5.1 is me. However, 4.3, 4.4, 4.5, 4.6, and 5.0 default to md. So when I upgrade all nodes in the cluster to 5.1, wrote some data, and finally rolled back to 4.6, the following error occurred:
ERROR 2023-09-11 07:12:46,448 [shard 0] database - Exception while populating keyspace 'alternator_ycsb' with column family 'ycsb' from file '/var/lib/scylla/data/alternator_ycsb/ycsb-1fa369f0506b11ee8364d7ed7565ea05': sstables::malformed_sstable_exception (invalid version for file me-4-big-Index.db. Name doesn't match any known version.)
ERROR 2023-09-11 07:12:46,451 [shard 0] database - Exception while populating keyspace 'system_distributed' with column family 'cdc_generation_timestamps' from file '/var/lib/scylla/data/system_distributed/cdc_generation_timestamps-fdf455c4cfec3e009719d7a45436c89d': sstables::malformed_sstable_exception (invalid version for file me-4-big-Index.db. Name doesn't match any known version.)
xxxxx
ERROR 2023-09-08 10:22:01,969 [shard 0] init - Startup failed: std::invalid_argument (Wrong sstable format name: me)
Although I have already recovered all tables in keyspace system and system_schema
Finally, I tried to clean up all the me format SStable. It still reports the following error：
2023-09-11T07:54:12.356630041Z ERROR 2023-09-11 07:54:12,356 [shard 0] init - Startup failed: std::runtime_error (Feature check failed. This node can not join the cluster because it does not understand the feature. Local node 100.71.8.124 features = {CDC, CDC_GENERATIONS_V2, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, CORRECT_STATIC_COMPACT_IN_MC, COUNTERS, DIGEST_FOR_NULL_VALUES, DIGEST_INSENSITIVE_TO_EXPIRY, DIGEST_MULTIPARTITION_READ, HINTED_HANDOFF_SEPARATE_CONNECTION, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, LWT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, MD_SSTABLE_FORMAT, NONFROZEN_UDTS, PER_TABLE_CACHING, PER_TABLE_PARTITIONERS, RANGE_SCAN_DATA_VARIANT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_TABLES_V3, STREAM_WITH_RPC_STREAM, TRUNCATION_TABLE, UDA, UNBOUNDED_RANGE_TOMBSTONES, VIEW_VIRTUAL_COLUMNS, WRITE_FAILURE_REPLY, XXHASH}, Remote common_features = {AGGREGATE_STORAGE_OPTIONS, CDC, CDC_GENERATIONS_V2, COMPUTED_COLUMNS, CORRECT_COUNTER_ORDER, CORRECT_IDX_TOKEN_IN_SECONDARY_INDEX, CORRECT_NON_COMPOUND_RANGE_TOMBSTONES, CORRECT_STATIC_COMPACT_IN_MC, COUNTERS, DIGEST_FOR_NULL_VALUES, DIGEST_INSENSITIVE_TO_EXPIRY, DIGEST_MULTIPARTITION_READ, EMPTY_REPLICA_PAGES, HINTED_HANDOFF_SEPARATE_CONNECTION, INDEXES, LARGE_PARTITIONS, LA_SSTABLE_FORMAT, LWT, MATERIALIZED_VIEWS, MC_SSTABLE_FORMAT, MD_SSTABLE_FORMAT, ME_SSTABLE_FORMAT, NONFROZEN_UDTS, PARALLELIZED_AGGREGATION, PER_TABLE_CACHING, PER_TABLE_PARTITIONERS, RANGE_SCAN_DATA_VARIANT, RANGE_TOMBSTONES, ROLES, ROW_LEVEL_REPAIR, SCHEMA_COMMITLOG, SCHEMA_TABLES_V3, SEPARATE_PAGE_SIZE_AND_SAFETY_LIMIT, STREAM_WITH_RPC_STREAM, TOMBSTONE_GC_OPTIONS, TRUNCATION_TABLE, TYPED_ERRORS_IN_READ_RPC, UDA, UDA_NATIVE_PARALLELIZED_AGGREGATION, UNBOUNDED_RANGE_TOMBSTONES, VIEW_VIRTUAL_COLUMNS, WRITE_FAILURE_REPLY, XXHASH})
The official document provides a rollback method from 5.1 to 5.0. How does it avoid SStable formatting errors from me to md? What is our upgrade order? From 4.6 to 5.0 and then to 5.1? Then perform a reverse rollback.
Thakns!