We are setting up scylladb cluster to be backed up in AWS S3 compatible storage and setup access_key, secet, endpoint in scylla-manager-agent.yaml correctly. From each node the command also works fine:
scylla-manager-agent check-location --debug --location s3:
But when we try and test backup from scyllamanger node it fails with:
Error: get backup target: location is not accessible
sctool backup -c ‘Scylla Prod 01’ -L ‘s3:’ --dry-run
NOTICE: this may take a while, we are performing disk size calculations on the nodes
2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded
2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded
…
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.365Z","N":"backup","M":"Generating backup target","cluster_id":"fe769e49-2639-4426-8829-77d5ccc20ae2","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Checking hosts connectivity","hosts":["2a0b:b580:0:0:xx:xxxx:xx:xx","22a0b:b580:0:0:xx:xxxx:xx:xx","2a0b:b580:0:0:xx:xxxx:xx:xx","2a0b:b580:0:0:xx:xxxx:xx:xx","2a0b:b580:0:0:xx:xxxx:xx:xx"],"_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Host check OK","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Host check OK","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Host check OK","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Host check OK","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Host check OK","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"cluster.client","M":"Done checking hosts connectivity","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:47 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:47.594Z","N":"backup","M":"Checking accessibility of remote locations","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:08:59 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:59.832Z","N":"cluster","M":"Creating new Scylla REST client","cluster_id":"fe769e49-2639-4426-8829-77d5ccc20ae2"}
May 08 13:08:59 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:08:59.884Z","N":"cluster.client","M":"Measuring datacenter latencies","dcs":["SCY-PRD1"]}
May 08 13:09:17 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:17.596Z","N":"cluster.client","M":"HTTP retry backoff","operation":"OperationsCheckPermissions","wait":"1s","error":"after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:29 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:29.832Z","N":"cluster","M":"Creating new Scylla REST client","cluster_id":"fe769e49-2639-4426-8829-77d5ccc20ae2"}
May 08 13:09:29 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:29.890Z","N":"cluster.client","M":"Measuring datacenter latencies","dcs":["SCY-PRD1"]}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Location check FAILED","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","location":"s3:<Bucket Name>","error":"giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Location check FAILED","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","location":"s3:<Bucket Name>","error":"giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Location check FAILED","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","location":"s3:<Bucket Name>","error":"giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Location check FAILED","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","location":"s3:<Bucket Name>","error":"giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Location check FAILED","host":"2a0b:b580:0:0:xx:xxxx:xx:xx","location":"s3:<Bucket Name>","error":"giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"backup","M":"Done checking accessibility of remote locations","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:48 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:48.597Z","N":"http","M":"GET /api/v1/cluster/Scylla%20Prod%2001/tasks/backup/target","from":"127.0.0.1:59140","status":400,"bytes":566,"duration":"61235ms","error":"get backup target: location is not accessible: 2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded; 2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded; 2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded; 2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded; 2a0b:b580:0:0:xx:xxxx:xx:xx: giving up after 2 attempts: after 30s: context deadline exceeded","_trace_id":"L9hVLdVmQBCl40aj9f0zkQ"}
May 08 13:09:59 prd-scylladbmanager scylla-manager[954628]: {"L":"INFO","T":"2023-05-08T13:09:59.832Z","N":"cluster","M":"Creating new Scylla REST client","cluster_id":"fe769e49-2639-4426-8829-77d5ccc20ae2"}