Memory issue for scylla-5.2.4-0.20230623.cebbf6c5df2b

Hi Team,

We are getting continuous errors for the memory for scylla 5.2.4. Please find the logs below

Dec 19 14:05:35 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Dec 19 14:05:39 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:05:39 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:05:45 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Dec 19 14:05:51 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:05:51 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:05:52 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system.batchlog: std::bad_alloc (std::bad_alloc)
Dec 19 14:05:52 ip-10-8-40-9 scylla: [shard 0] batchlog_manager - Exception in batch replay: exceptions::read_failure_exception (Operation failed for system.batchlog - received 0 responses and 1 failures from 1 CL=ONE.)
Dec 19 14:05:55 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:00 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:00 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:06:00 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:00 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:06:05 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:06 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:06 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:06:15 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:21 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:21 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:06:24 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_auth.roles: std::bad_alloc (std::bad_alloc)
Dec 19 14:06:24 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.129, to read from system_auth.roles: std::bad_alloc
Dec 19 14:06:25 ip-10-8-40-9 scylla: [shard 0] storage_proxy - Exception when communicating with 10.8.40.9, to read from system_distributed.service_levels: std::bad_alloc (std::bad_alloc)

Please help us in rectifying the problem.

Regards,
Soyal Badkur

We would require more details. A backtrace would be helpful.
Does the system crash? If so please send the coredump.
Are you using large collections?
Please see the documentation for reporting a problem. You can open a support ticket if you are a customer and a GitHub issue otherwise.