Restore data: initialize versioned SSTables: not a Scylla Manager snapshot

ScyllaDB
, , ,
1

*Installation details
#ScyllaDB version: 5.0.13
#Cluster size: 3 nodes
os (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu

restore data: initialize versioned SSTables: not a Scylla Manager snapshot
despite the snapshot tag is exact same.

What am I missing? How to resolve this?

2

Please share more details, the exact steps you took, the full error message, and any other details you have.

3

Hello Guy,
Thank you for your reply and help -

Here is the summary what we did -

For Backup -

  1. Started Scylla Manager on monitoring stack machine
  2. Started Scylla-mananger-agent on all 3 nodes of scylla-cluster.
  3. Added cluster to scylla manager.
  4. started backup using sctool every 6 hours and retention last 2 to s3 bucket.

For restore

  1. Created a new scylla cluster and also created new manager machine.
  2. Started Scylla Manager on monitoring stack machine
  3. Started Scylla-mananger-agent on all 3 nodes.
  4. Added cluster to scylla manager.
  5. Tried restoring using the snapshot tag, but facing error even though the tag is same.

Run: 818902-de0b-11ef-91h8-02d6b68b54cd
Status: ERROR
Cause: restore data: initialize versioned SSTables: not a Scylla Manager snapshot tag, expected format is sm_20060102150405UTC
Start time: 24 Jan 25 04:27:21 UTC
End time: 24 Jan 25 04:27:22 UTC
Duration: 0s
Progress: 0% | 0%
Snapshot Tag: sm_20250123110656UTC

4

From the information pasted above it looks like sm_20250123110656UTC has a correct snapshot tag format.

Please provide Scylla Manager logs from the time of running the restore.

1 Like
5

Hello @Michal_Trzonkowski,
Thank you for your reply,
Below are the logs -

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.052Z”,“N”:“scheduler”,“M”:“PutTask”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“schedule”:{“cron”:“”,“window”:null,“timezone”:“Etc/UTC”,“start_date”:“0001-01-01T00:00:00Z”,“interval”:“”,“num_retries”:3,“retry_wait”:“10m”},“properties”:{“keyspace”:[“myredactedkeyspace”],“location”:[“s3:my-reducted-s3-bucket”],“restore_tables”:true,“snapshot_tag”:“sm_20250121065957UTC”},“create”:true,“_trace_id”:“jwRC1_rlT4OgDEFa8h4scg”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.055Z”,“N”:“scheduler.136499a1”,“M”:“Schedule”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“in”:“0s”,“begin”:“2025-01-23T10:37:18.055Z”,“retry”:0,“_trace_id”:“jwRC1_rlT4OgDEFa8h4scg”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.056Z”,“N”:“scheduler.136499a1”,“M”:“Run started”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“retry”:0,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.265Z”,“N”:“backup”,“M”:“Restore”,“cluster_id”:“136499a1-313a-4d2c-b090-e9945efe8ad8”,“task_id”:“25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“run_id”:“04bbd675-d976-11ef-804e-0237f91a5b9b”,“target”:{“location”:[“s3:my-reducted-s3-bucket”],“keyspace”:[“myredactedkeyspace”,“!system”,“!system_schema”,“!system_distributed_everywhere.cdc_generation_descriptions_v2”,“!system_distributed.cdc_streams_descriptions_v2”,“!system_distributed.cdc_generation_timestamps”,“!._scylla_cdc_log”],“snapshot_tag”:“sm_20250121065957UTC”,“batch_size”:2,“restore_tables”:true,“continue”:true},“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.434Z”,“N”:“backup.restore”,“M”:“Created restore units”,“units”:[{“Keyspace”:“myredactedkeyspace”,“Size”:711636847007,“Tables”:[{“Table”:“myreductedtable”,“Size”:388144379774},{“Table”:“myreductedtable”,“Size”:323492467233}]}],“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.436Z”,“N”:“backup.restore”,“M”:“Awaiting schema agreement…”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.443Z”,“N”:“backup.restore”,“M”:“Done awaiting schema agreement”,“duration”:“6.811967ms”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.542Z”,“N”:“backup.restore”,“M”:“Started restoring tables”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:18.542Z”,“N”:“backup.restore”,“M”:“Disabling table’s gc_grace_seconds”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:19 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:19.239Z”,“N”:“backup.restore”,“M”:“Disabling table’s gc_grace_seconds”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.128Z”,“N”:“backup.restore”,“M”:“Awaiting schema agreement…”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.136Z”,“N”:“backup.restore”,“M”:“Done awaiting schema agreement”,“duration”:“7.631435ms”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.136Z”,“N”:“backup.restore”,“M”:“Restoring location”,“location”:“s3:my-reducted-s3-bucket”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.394Z”,“N”:“backup.restore”,“M”:“Initialized restore hosts”,“hosts”:[{“Host”:“172.31.20.203”,“OngoingRunProgress”:null},{“Host”:“172.31.20.72”,“OngoingRunProgress”:null},{“Host”:“172.31.27.215”,“OngoingRunProgress”:null}],“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.444Z”,“N”:“backup.restore”,“M”:“Restoring manifest”,“manifest”:{“Location”:“s3:my-reducted-s3-bucket”,“DC”:“datacenter1”,“ClusterID”:“46s32eg7-9013-45b7-a59b-f071cot3jpae”,“NodeID”:“80ba459f-2500-4f96-a8b0-69ea500b0848”,“TaskID”:“7e1d88ad-0dc8-455c-ab94-42fd3e6619ae”,“SnapshotTag”:“sm_20250121065957UTC”,“Temporary”:false},“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.445Z”,“N”:“backup.restore”,“M”:“Restoring table”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.447Z”,“N”:“backup.restore”,“M”:“Initialized SSTable bundle pool”,“sstable_ids”:[“3631”,“4413”,“4478”,“719”,“4452”,“1017”,“4030”,“4171”,“4356”,“4249”,“4387”,“4417”,“4355”,“4430”,“4486”,“652”,“2555”,“3173”,“4383”,“4456”,“673”,“718”,“4373”,“4424”,“4466”,“4482”,“4224”,“4295”,“4401”,“4421”],“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.450Z”,“N”:“backup.restore”,“M”:“Received table’s version”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“version”:“efaebc80d97211efb40d5d352ab1f974”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.450Z”,“N”:“backup.restore”,“M”:“Found table’s source and destination directory”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“src_dir”:“s3:my-reducted-s3-bucket/backup/sst/cluster/46s32eg7-9013-45b7-a59b-f071cot3jpae/dc/datacenter1/node/80ba459f-2500-4f96-a8b0-69ea500b0848/keyspace/myredactedkeyspace/table/myreductedtable/1a600df0fa2511ed9a140dd3a1cdb004”,“dst_dir”:“data:myredactedkeyspace/myreductedtable-efaebc80d97211efb40d5d352ab1f974/upload”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.500Z”,“N”:“backup.restore”,“M”:“Restoring table finished”,“keyspace”:“myredactedkeyspace”,“table”:“myreductedtable”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.500Z”,“N”:“backup.restore”,“M”:“Restoring manifest finished”,“manifest”:{“Location”:“s3:my-reducted-s3-bucket”,“DC”:“datacenter1”,“ClusterID”:“46s32eg7-9013-45b7-a59b-f071cot3jpae”,“NodeID”:“80ba459f-2500-4f96-a8b0-69ea500b0848”,“TaskID”:“7e1d88ad-0dc8-455c-ab94-42fd3e6619ae”,“SnapshotTag”:“sm_20250121065957UTC”,“Temporary”:false},“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.500Z”,“N”:“backup.restore”,“M”:“Restoring location finished”,“location”:“s3:my-reducted-s3-bucket”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.500Z”,“N”:“backup.restore”,“M”:“Restoring tables finished”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.510Z”,“N”:“scheduler.136499a1”,“M”:“Run ended with ERROR”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“status”:“ERROR”,“cause”:“restore data: not restored bundles [3631 4413 4478 719 4452 1017 4030 4171 4356 4249 4387 4417 4355 4430 4486 652 2555 3173 4383 4456 673 718 4373 4424 4466 4482 4224 4295 4401 4421]: initialize versioned SSTables: not a Scylla Manager snapshot tag, expected format is sm_20060102150405UTC”,“duration”:“2.444916447s”,“_trace_id”:“93PMHTMNRLClxnK6fQG5xA”}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.510Z”,“N”:“scheduler.136499a1”,“M”:“Retry backoff”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“backoff”:“10m0s”,“retry”:1}

Jan 23 10:37:20 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:20.510Z”,“N”:“scheduler.136499a1”,“M”:“Schedule”,“task”:“restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“in”:“9m59s”,“begin”:“2025-01-23T10:47:20.510Z”,“retry”:1}

Jan 23 10:37:34 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:34.394Z”,“N”:“http”,“M”:“GET /api/v1/cluster/test_indhan/task/restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“from”:“127.0.0.1:45462”,“status”:200,“bytes”:553,“duration”:“26ms”,“_trace_id”:“fqBU2r3oR0isqU1VSDWoXw”}

Jan 23 10:37:34 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:37:34.442Z”,“N”:“http”,“M”:“GET /api/v1/cluster/test_indhan/task/restore/25132a36-9f31-40c9-b7e5-fa5d020ff0fe/latest”,“from”:“127.0.0.1:45462”,“status”:200,“bytes”:1164,“duration”:“47ms”,“_trace_id”:“Gj8_DhBhQyevo8oukRNz0Q”}

Jan 23 10:38:18 scylla-manager[2100920]: {“L”:“INFO”,“T”:“2025-01-23T10:38:18.115Z”,“N”:“scheduler”,“M”:“Task deleted”,“cluster_id”:“136499a1-313a-4d2c-b090-e9945efe8ad8”,“task_type”:“restore”,“task_id”:“25132a36-9f31-40c9-b7e5-fa5d020ff0fe”,“_trace_id”:“kymEDapqRIWArT5iijM0sw”}