Hello!
I have old scylla cluster v4.3. It started from 5 nodes, then 6 and about 2 months ago i added 7th node. It worked fine untill i added 7th node. Now all nodes sometimes send error:
Aug 30 14:15:04 scylla1 scylla[3433932]: Reactor stalled for 525 ms on shard 1.
…
Aug 30 14:15:04 scylla1 scylla[3433932]: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
Aug 30 14:15:04 scylla1 scylla[3433932]: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912
or
Aug 30 14:18:19 scylla1 scylla[3433932]: Reactor stalled for 526 ms on shard 3.
…
Aug 30 14:18:19 scylla1 scylla[3433932]: /opt/scylladb/libreloc/libpthread.so.0+0x0000000000009431
Aug 30 14:18:19 scylla1 scylla[3433932]: /opt/scylladb/libreloc/libc.so.6+0x0000000000101912
with different shards and different backtraces
I’m not shure if it is related to keyspaces/tables damage, or incorrect scylla configuration or network issues or scylla bug. Tried repair tables and drain/stop new added node - it didn’t help.