Unexpected exception when pinging

ScyllaDB
,
1

Hey,
Recently I’ve deployed many scylla nodes on AWS but I’m constantly running into WARN direct_failure_detector - unexpected exception when pinging a84dc681-ddee-4c0e-b140-4c686c1d7297: seastar::rpc::unknown_verb_error (unknown verb) (for many different hosts) which on some nodes result in shutdown and on others just spams.
Has anyone else encountered this?

Thanks

2

Thank you for reporting this, it is likely an issue we need to fix. That said, it might be better to move this discussion to the ScyllaDB issue tracker, can you please open a new issue and add the following details (in addition to your description above):
What ScyllaDB version are you running? Are all nodes running the same version?
Do you see any other errors in the logs?

3

Thanks for the quick reply!
All my nodes (~100) are running version 5.2.9, with lightweight resources: 2Gi of memory and 2 vCPUs. I know it might be weird but that’s to simulate a very specific workload…
I also saw multiples of:

  • WARN raft_group_registry - Raft server id 0003c518-4d07-4bb2-a3a3-3d5e047c550d cannot be translated to an IP address.
  • WARN storage_proxy - Exception when communicating with 172.23.5.27, to read from system_distributed.service_levels: std::bad_alloc.

I suspect it’s some kind of resource problem as I am trying to have them really slim, but didn’t saw something indicative of that.

4

@kbr you might want to take a look at this, AFAIK you were chasing this issue with IP address translation.
We also need to get to the bottom of this unknown verb error.

Like I said above, it is best to open an issue in the ScyllaDB issue tracker because the forum is not the best venue for investigating problems.

You can use this link to open an issue: https://github.com/scylladb/scylladb/issues/new

1 Like