Scylla-manager backup failure

Hello All,

A backup to azure from scylladb manager fails when we run a dry run backup using “sctool backup -c Cluster -L azure:scylladb-backup --dry-run”.

But when we run the check location on each scylla cluster node with
scylla-manager-agent check-location --location azure:scylladb-backup it wont return any error.
The same thing happens for scylla-manager-agent check-location --debug --location azure:scylla-db

what we have done so far;

  1. check location on all cluster nodes using the check location command with debug and they all returned without any error!

  2. the check location with debug on all cluster nodes returned successfully writing a temp file to the azure location and deleting them.

The issue I am having now is when the command from scylladb manager is executed “sctool backup -c Cluster -L azure:scylladb-backup --dry-run”, we get the error below.

"we get the error below: Jun 13 08:55:52 scylladb scylla-manager[1957]: {“L”:“ERROR”,“T”:“2024-06-13T08:55:52.685Z”,“N”:“backup”,“M”:“Failed to access location from node”,“node”:“XXX.XX.XX.XX”,“location”:“azure:scylladb-backup”,“error”:“17X.XX.XX.XX: giving up after 2 attempts: after 30s: context deadline”

Kindly help with steps or ways to troubleshoot and resolve this issue.

Hi @Mike ,

scylla-manager-agent reads the configuration directly from /etc/scylla-manager-agent/scylla-manager-agent.yaml when it’s executed. Therefore, any changes you make to the configuration file are automatically reflected when you call the scylla-manager-agent CLI.

The Scylla Manager server performs location checks through the agent running as a service on the node. Whenever you make changes to scylla-manager-agent.yaml, you must restart the service to apply the recent configuration changes.

Have you updated the location in the .yaml file but haven’t restarted the service yet? Please try restarting the scylla-manager-agent.service on all nodes.

sudo systemctl restart scylla-manager-agent

Br,

1 Like