I am running a single self-managed Scylla instance in AWS. After running 1 week without any problems, I got this message from AWS:
EC2 has detected degradation of the underlying hardware hosting your Amazon EC2 instance (instance-ID: i-XX).
I needed to stop-and-start my instance. After spinning the instance up again and sshed to the instance, I see it fails to boot because of this error:
Failed mounting RAID volume!
Scylla has aborted startup because of a missing RAID volume.
Is there anything I can do to restart scylla with the same dataset as before the restart?
As we spoke in Users Slack, no – your data is gone, particularly if you are using locally attached SSDs:
The data on an SSD instance volume persists only for the life of its associated instance.
SSD instance store volumes - Amazon Elastic Compute Cloud
Thanks @felipemendes, here is the discussion from the Slack channel:
Nope, AWS allocated you a new set of local SSDs. You can recreate the array and then start the replace node procedure.
You can recreate the array and then start the replace node procedure. what do you mean by that? So
the data is lost?
in that particular node, yes. You do have other replicas, no?
No, no other replica’s atm, it’s just for development though
Ok and do you have any reference to You can recreate the array and then start the replace node procedure.?
https://opensource.docs.scylladb.com/stable/operating-scylla/procedures/cluster-management/rebuild-node.htmlhttps://opensource.docs.scylladb.com/stable/kb/raid-device.htmlDepending on your ScyllaDB version you may need to manually delete the systemd
.mount resources in your /etc and follow with a
systemctl daemon-reload afterwards.