Backup failing up giving indexing file error

ScyllaDB
,
1

Installation details
Running a 3 node scylla cluster in GKE.
#ScyllaDB version: 6.2.0
#Cluster size: 3
os (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu

Hello guys,
Using scylla manager for scheduled backup. After some successful backup iterations, it gets failed showing indexing file error

Screenshot 2025-03-06 at 10.14.27 AM
Screenshot 2025-03-06 at 10.14.27 AM1988Ă—346 19 KB

2

Looking at your error, the backup is failing because Scylla Manager can’t find any files to index on node 10.240.5.5. Since this worked before and now suddenly fails, here’s what I’d check:

First, verify if there’s actually data to backup:

  • SSH into the failing node (10.240.5.5) and check /var/lib/scylla/data/
  • Look for your keyspace directories and see if there are any .db files (SSTables)
  • Run du -sh /var/lib/scylla/data/your_keyspace_name/ to see actual data size
  • If it shows 0 bytes or the directory is missing, that’s your problem

Check what happened to your data:

  • Did someone accidentally truncate or drop tables recently?
  • Was there a major compaction that might have removed all data?
  • Check if this specific node is properly part of the cluster: nodetool status
  • Verify the node can see the keyspaces: cqlsh -e "DESC KEYSPACES"

Verify backup configuration:

  • Double-check your backup task configuration - make sure it’s targeting the right keyspaces/tables
  • Run sctool task list to see your backup tasks
  • Compare the working vs failing backup configurations
1 Like
3

This is the logs of scylla-manger
{“L”:“ERROR”, “M”:“Indexing snapshot files failed on host”, “N”:“backup.index”, “S”:“github/scylladb/go-log.Logger.log
github/scylladb/go-log@v0.0.7/logger.go:101
github/scylladb/go-log.Logger.Error
github/scylladb/go-log@v0.0.7/logger.go:84
github/scylladb/scylla-manager/v3/pkg/service/backup.(*worker).Index.func2
githubcom/scylladb/scylla-manager/v3/pkg/service/backup/worker_index.go:33
githubcom/scylladb/scylla-manager/v3/pkg/service/backup.hostsInParallel.func2
githubcom/scylladb/scylla-manager/v3/pkg/service/backup/parallel.go:89
githubcom/scylladb/scylla-manager/v3/pkg/util/parallel.Run.func1
githubcom/scylladb/scylla-manager/v3/pkg/util@v0.0.0-20241104134613-aba35605c28b/parallel/parallel.go:79”, “T”:“2025-06-22T06:00:08.273Z”, “_trace_id”:“UORGpSjeQWKBAeP43xVHKw”, “error”:“10.244.11.91: could not find any files”, “errorStack”:"githubcom/scylladb/scylla-manager/v3/pkg/service/backup.hostsInParallel.func1
githubcom/scylladb/scylla-manager/v3/pkg/service/backup/parallel.go:86
githubcom/scylladb/scylla-manager/v3/pkg/util/parallel.Run.func1
githubcom/scylladb/scylla-manager/v3/pkg/util@v0.0.0-20241104134613-aba35605c28b/parallel/parallel.go:72
runtime.goexit
runtime/asm_amd64.s:1700
", “host”:“10.244.11.91”}

Backup’s coming properly after reschedule until this disaster comes.