Originally from the User Slack
@SabinSabin**:** I have really old Scylla Server (Scylla version 4.2.1-0.20201108.4fb8ebccff) which is “running” at work. Recently, I have encountered issues where there’s random segfaults on shard X. How can I debug and possibly fix it? I want to migrate it to newer version but this issue is holding me back
I am mostly seeing these on the journalctl for Scylla Server
Oct 22 04:53:29 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: [shard 0] compaction - Compacting [/var/lib/scylla/data/system/clients-ca0f635d863036098d93a1fc06f2a5e5/mc-12922-big-Data.db:level=0, /var/lib/scylla/data/system/clients-ca0f635d863036098d93a1fc06f2a5e5/mc-12936-big-Data.db:level=0, ]
Oct 22 04:53:30 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: [shard 0] compaction - Compacted 2 sstables to [/var/lib/scylla/data/system/clients-ca0f635d863036098d93a1fc06f2a5e5/mc-12950-big-Data.db:level=0, ]. 93kB to 17kB (~18% of original)
in 118ms = 144kB/s. ~256 total partitions merged to 1.
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: Segmentation fault on shard 0.
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: Backtrace:
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002ef3122
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e976a0
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e97945
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e979e0
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x00007f548046ca8f
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000f2b223
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000f2b61f
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001087aea
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000fe2e23
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001000ab5
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001005c1b
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001007949
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x000000000100807b
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001007a5c
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x000000000116f281
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x000000000117171d
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001171c9a
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x00000000010d0f97
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001089760
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000001092a23
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x00000000010c2f22
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000e380a9
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000e3ac0c
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e2b3e6
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e94a47
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e94dbe
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002ecbe9d
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e28c7a
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000002e2932e
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000da8b40
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: /opt/scylladb/libreloc/libc.so.6+0x0000000000027041
Oct 22 04:53:39 <http://scylladb-1.mycompany.com|scylladb-1.mycompany.com> scylla[1841]: 0x0000000000cc4a2d
This crashes the service and the kernel on… Ubuntu 18.04 TL@avi
@avi**:** It is highly recommended to upgrade both the server OS and the ScyllaDB soft@Sabinare
@Sabin**:** If I create a new instance with recent version of ScyllaDB, would it be able to join the@avicluster?
@avi**:** It’s not recommended, especially for such old versions
Upgrade in place, one version at a time, until you reach a supported version, and don’t let it lag i@Sabin the future
@Sabin**:** I am worried about the data. Also, I don’t have much experience wi@avih the DB itself.
@avi**:** Take a backup.
Leaving it like that will cause it to rot until one day yo@Sabin cannot recover it.
@Sabin**:** Is there a way to fix this error apart from upgrading? I have backup but recoveri@avig it takes way too long.
@avi**:** Try to decode it on @Sabintp://backtrace.scylladb.com
@Sabin**:** I wonder if I can do rolling up@avirade of ScyllaDB from 4.x to 6.x
@avi**:** No, we only test one minor version at a time
@Sabin**:** That will take time. The node is now up as the actual issue was with hardware (faulty RAM sticks). I will try to get the data off the nodes and try to restore it on newer version on another cluster.