UnavailableException (x required but only 0 alive)

Hi to everyone,
We are trying to migrate from cassandra to scylla 4.6 version.
For scylla we have a cluster with 4 nodes.
Memory : 100 GB per instance
Disk : 1Tb
Cpu pinning has been done.
Compaction strategy for all tables as of now : Size Tiered.
RF is 4. Since all nodes in same data centre we are using SimpleSnitch.
com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_ONE (1 required but only 0 alive)
Getting this error intermittently rarely but causes a lot of write errors in that time. Scylla yaml also attached. Grafana screenshots attached for that time with all node view.
What could be the causes for this error, if all nodes are up and RF is also 4?

cluster_name: '<cluster_name>'
num_tokens: 256
workdir: /data/scylla
# Directory where Scylla should store data on disk.
   - /data/scylla/data

commitlog_directory: /data/scylla/commitlog

commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000

commitlog_segment_size_in_mb: 32

# seeds address(es) are mandatory!
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
          # seeds is actually a comma-delimited list of addresses.
          # Ex: "<ip1>,<ip2>,<ip3>"
          - seeds: <seeds>
# Setting listen_address to is always wrong.
listen_address: <ip>

# broadcast_address:

# Ignore this property if the network configuration automatically routes between the public and private networks such as EC2.
# listen_on_broadcast_address: false

native_transport_port: 9042

#native_transport_port_ssl: 9142

read_request_timeout_in_ms: 5000

write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000

endpoint_snitch: SimpleSnitch

rpc_address: <rpc address>
# rpc_interface: eth1
# rpc_interface_prefer_ipv6: false

# port for Thrift to listen for clients on
rpc_port: 9160

# port for REST API server
api_port: 10000

# IP for the REST API server
batch_size_warn_threshold_in_kb: 5
batch_size_fail_threshold_in_kb: 50
#developer_mode: true
murmur3_partitioner_ignore_msb_bits: 12
api_ui_dir: /opt/scylladb/swagger-ui/dist/
api_doc_dir: /opt/scylladb/api/api-doc/
authenticator: PasswordAuthenticator
authorizer: CassandraAuthorizer

Scylla 4.6 is quite an old release at this point. It is possible that the problem you are suffering from was fixed in more current ScyllaDB releases. Please upgrade to 5.2 (or even better, 5.4) and report back whether the problem persists.


Thanks @Botond_Denes , will upgrade and replicate

I see you’re using PasswordAuthenticator, please also make sure to alter RF accordingly, it may cause similar issues with default RF (see Enable Authentication | ScyllaDB Docs)