Upgrading ScyllaDB version, downtime, and Raft configuration

[Guy]

Originally from the User Slack

@Carter: unless i’m misunderstanding, you cannot upgrade from open source 5.2 to 5.4 without downtime for the raft upgrade?

@Felipe_Cardeneti_Mendes: You’re misunderstanding. Unless you strictly opted out of Raft in the configuration, Raft will be enabled as you upgrade. All you need to check is whether it completed successfully.

@Carter: I tried the upgrade on staging – just changing our image version from 5.2 to 5.4, no config changes.

It required all nodes to be upgraded and online before the upgrade could commence, and only then, it would open the port for CQL.

consistent_cluster_management was not set to true nor false when I did the upgrade – is that why?
I should mention that our clusters are tiny at 3 nodes each

@Felipe_Cardeneti_Mendes: Well, yes all nodes must be up and running before you start the upgrade. Not sure WDYM on the last statement for CQL. Are you saying you upgraded and the nodes wouldn’t get start CQL until you finalized the entire upgrade? That shouldn’t happen.

@Carter: Hmm, I stand corrected! I double checked my logs and the way I was streaming them, it was coming from all pods and it caused some confusion. Thanks for your help, @Felipe_Leme_Gasparini!

[Yousuf_Jawwad]

Error with upgrading …

INFO  2024-09-12 17:07:52,103 [shard 0:strm] raft_group0 - finish_setup_after_join: group 0 ID present, loading server info.                                                                                    │
│ INFO  2024-09-12 17:07:52,103 [shard 0:strm] raft_group0 - finish_setup_after_join: SUPPORTS_RAFT feature enabled. Starting internal upgrade-to-raft procedure.                                                 │
│ INFO  2024-09-12 17:07:52,103 [shard 0:strm] raft_group0_upgrade - Already upgraded.                                                                                                                            │
│ INFO  2024-09-12 17:07:52,103 [shard 0:strm] cdc - Latest generation seen during startup: (2024/05/05 12:05:28, 140eef67-8a99-4233-8162-48124267486f)                                                           │
│ INFO  2024-09-12 17:07:52,112 [shard 0:strm] cdc - Starting to use generation (2024/05/05 12:05:28, 140eef67-8a99-4233-8162-48124267486f)                                                                       │
│ INFO  2024-09-12 17:07:52,117 [shard 0:strm] cdc - Generation (2024/05/05 12:05:28, 140eef67-8a99-4233-8162-48124267486f): streams description table already updated.                                           │
│ INFO  2024-09-12 17:07:52,117 [shard 0:strm] raft_topology - The cluster is ready to start upgrade to the raft topology. The procedure needs to be manually triggered. Refer to the documentation               │
│ INFO  2024-09-12 17:07:52,118 [shard 0:main] init - starting tracing                                                                                                                                            │
│ INFO  2024-09-12 17:07:52,118 [shard 0:main] init - SSTable data integrity checker is disabled.                                                                                                                 │
│ INFO  2024-09-12 17:07:52,118 [shard 0:main] service_level_controller - start_legacy_update_from_distributed_data: starting configuration polling loop                                                          │
│ INFO  2024-09-12 17:07:52,119 [shard 0:main] init - starting auth service                                                                                       

I have followed the instructions from here i.e. applied changes to scylla.yaml before restart and then restarted the nodes.

What am I missing?