Workaround for compatibility issue, backup and restore using an older ScyllaDB version

[Guy]

Originally from the User Slack

@Sabin: Hi Team,
After restoring the backup on entire different cluster using the script (https://github.com/GoogleCloudPlatform/cassandra-cloud-backup), we noticed that the cluster status is DN as it picked up IP address of the previous cluster which is not accessible from this newly created cluster where restore operation is being done. Is there way to change the IP address somehow?
The script that is used backs up entire data directory along with commit logs and saved caches directory for each node of the cluster.
Old cluster and it’s IP looks like this:

│    │ CQL        │ REST      │ Address       │ Uptime       │ CPUs │ Memory    │ Scylla │ Agent │ Host ID                              │
├────┼────────────┼───────────┼───────────────┼──────────────┼──────┼───────────┼────────┼───────┼──────────────────────────────────────┤
│ UN │ DOWN (0ms) │ UP (0ms)  │ 192.168.100.1 │ 8942h15m34s  │ 16   │ 125.82GiB │ 4.2.1  │ 2.2.0 │ d5e731f8-db6f-4d26-8948-04802c88de85 │
│ UN │ DOWN (0ms) │ UP (0ms)  │ 192.168.100.3 │ 15242h54m51s │ 16   │ 125.82GiB │ 4.2.1  │ 2.2.0 │ c67857a9-16ae-4f5e-ac68-0e2f36dcb8d3 │
│ UN │ DOWN (0ms) │ UP (25ms) │ 192.168.100.4 │ 15243h9m6s   │ 16   │ 125.82GiB │ 4.2.1  │ 2.2.0 │ 4d347c78-bb94-4224-87eb-2357a1706d2d │

GitHub: GitHub - GoogleCloudPlatform/cassandra-cloud-backup: Cassandra backups to Google Cloud Storage

@Chaitanya_Tondlekar: Better to use scylla manager 3.2 for better and easy data restoratiom

@Sabin: We already have Scylla Manager v2.2.0-0.20201103.a3fdb862. I could configure it to push backup data to S3 but I am not sure if same data can be restored on newer cluster.
There’s already an issue with backup/restore for they take way too much time (400 GB in 6 hours) to complete

@Chaitanya_Tondlekar: 1. upgrade the scylla manager to 3.2 atleast.
2. If not, you can use https://github.com/scylladb/scylla-manager/tree/master/ansible/restore

GitHub: scylla-manager/ansible/restore at master · scylladb/scylla-manager

@Sabin: how is the compatibility? Scylla version itself is 4.2.1, will Scylla Manager function with that version?

@Chaitanya_Tondlekar: you can check the compalibility but ansible should work if you have taken backup from scylla manager

@Sabin: Sadly it is not from Scylla Manager. It’s backed up using the script I mentioned above, which is why the issue

@avi: ScyllaDB 4.2.1 has reached end-of-life, upgrade to a supported version

@Sabin: It partly for that purpose @avi Backup is weird and we inherited the system. ScyllaDB’s knowledge isn’t that much in our case. So I am just trying to figure out what we can do.

@avi: I guess you can take snapshots and copy the files away manually, if you can’t find your arms and legs

@Sabin: Can’t I just copy the entire data directory and restore it on new cluster?

@avi: You can restore it with nodetool refresh --load-and-stream

@Sabin: okay. I will give it a try once more and report back

[sprj]

Apart from the approach listed above, I gave it a go with Scylla Manager as well. On old cluster running version 4.2.1 of ScyllaDB, after configuring Wasabi bucket to store backups and verifying that the setup works, I took a backup of a keyspace (productdb) via Scylla Manager (v2.2) using following command:

sctool backup -c scprod -L 's3:scbackup' -K 'productdb'

I, then, used DESCRIBE KEYSPACE productdb query via cqlsh to get the schema and stored it for later use.

On new cluster running ScyllaDB v6.0.1, and Scylla Manager v3.3, I set the Wasabi bucket to the same bucket as with old cluster. I SOURCEed the schema dumped previously to prepare for restore. Afterwards, I ran the following command to restore the keyspace

sctool restore -c test-cluster -L 's3:scbackup' -T sm_20240724121252UTC --restore-tables 

It went well for 4% then it failed. On one of the node, I saw following error:

Jul 24 10:08:02 sctest-01.private.example.com scylla[10365]:  [shard 0:strm] sstables_loader - load_and_stream: ops_uuid=1aa26155-8445-4b1c-9939-2baeb0afb438, ks=productdb, table=product, send_ph
ase, err=sstables::malformed_sstable_exception (Failed to read partition from SSTable /var/lib/scylla/data/productdb/product-a7eb7e6049a311efa338c715591e0272/upload/mc-301908-big-Data.db due to Colu
mn cvr missing in current schema)
Jul 24 10:08:02 sctest-01.private.example.com scylla[10365]:  [shard 0:strm] stream_session - [Stream #1aa26155-8445-4b1c-9939-2baeb0afb438] Failed to handle STREAM_MUTATION_FRAGMENTS (receive and distribute p
hase) for ks=productdb, cf=product, peer=10.1.169.163: seastar::nested_exception: std::runtime_error (Sender failed) (while cleaning up after std::runtime_error (Sender failed))
Jul 24 10:08:02 sctest-01.private.example.com scylla[10365]:  [shard 0:strm] sstables_loader - send_meta_data: failed to process source from node=10.1.169.163, err=std::runtime_error (send_meta_data: got error
 code=-1 from node=10.1.169.163)
Jul 24 10:08:02 sctest-01.private.example.com scylla[10365]:  [shard 0:strm] sstables_loader - load_and_stream: ops_uuid=1aa26155-8445-4b1c-9939-2baeb0afb438, ks=productdb, table=product, finish_
phase, err=std::runtime_error (send_meta_data: got error code=-1 from node=10.1.169.163)

I don’t know what to try next or where to look.
Entire idea was to at least have a backup which can be restored so that we could migrate to newer cluster avoiding potential disaster of data loss.

[felipemendes]

@sprj

I don’t know what to try next or where to look.

It looks like the error points to a different schema than the one existing in the backup. In this case, column cvr is missing. You should restore to the same schema as the source SSTables expect. IIRC, back in SM 2.2 days we used to dump the schema cql alongside the SSTables.