Can't get host id to ip mapping error

ScyllaDB
,
1

Hello, I am new to scylladb. I set up all the prerequisite on gke using GitOps (kubectl) | ScyllaDB Docs and all the component are up and running. Now I am following the instruction on ScyllaClusters | ScyllaDB Docs to deploy a cluster. Only the difference is the region names (I am using us-central-1 instead of us-east-1) and the cluster/dc name. The pod for the first region started without issue and the second region one did not pass the readiness

> kubectl -n scylla-cluster get pod
NAME                                     READY   STATUS    RESTARTS   AGE
testscylla-testscylla1-us-central1-a-0   4/4     Running   0          17h
testscylla-testscylla1-us-central1-b-0   3/4     Running   0          93m

> kubectl -n scylla-cluster describe pod testscylla-testscylla1-us-central1-b-0
Name:             testscylla-testscylla1-us-central1-b-0
Namespace:        scylla-cluster
Priority:         0
Service Account:  testscylla-member
Node:             gke-sbx-202509-scylladb-n2ds2-c3b17c58-4d0u/10.128.0.24
Start Time:       Tue, 28 Oct 2025 08:52:34 -0400
Labels:           app=scylla
                  app.kubernetes.io/managed-by=scylla-operator
                  app.kubernetes.io/name=scylla
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=testscylla-testscylla1-us-central1-b-7bcc974b68
                  scylla/cluster=testscylla
                  scylla/datacenter=testscylla1
                  scylla/rack=us-central1-b
                  scylla/rack-ordinal=1
                  scylla/scylla-version=2025.1.5
                  statefulset.kubernetes.io/pod-name=testscylla-testscylla1-us-central1-b-0
Annotations:      internal.scylla-operator.scylladb.com/sysctls: ["fs.aio-max-nr=30000000"]
                  prometheus.io/port: 9180
                  prometheus.io/scrape: true
                  scylla-operator.scylladb.com/inputs-hash: zH+F33phgDxNbW6LFjiInXOTqlWbThr3SiCEmtdWR9Xdh94e88g58ynChaT6f9FqetXtcqugAgRJUMfZYYetWw==
Status:           Running
IP:               10.130.6.15
IPs:
  IP:           10.130.6.15
Controlled By:  StatefulSet/testscylla-testscylla1-us-central1-b
...
...
Containers:
  scylla:
    Container ID:  containerd://7f3ee4eeb19194c7734ab268d9cb411d5b1d00861b4901f86891f6c2c68a7d6c
    Image:         docker.io/scylladb/scylla:2025.1.5
    Image ID:      docker.io/scylladb/scylla@sha256:acd61debf250089e926db5c6a0d4675d42324feadec3965c04f9b6a7f7679216
    Ports:         7000/TCP, 7001/TCP, 9042/TCP, 9142/TCP, 7199/TCP, 9180/TCP, 9100/TCP, 9160/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      /usr/bin/bash
      -euEo
      pipefail
      -O
      inherit_errexit
      -c
      trap 'kill $( jobs -p ); exit 0' TERM
      
      printf 'INFO %s ignition - Waiting for /mnt/shared/ignition.done\n' "$( date '+%Y-%m-%d %H:%M:%S,%3N' )" > /dev/stderr
      until [[ -f "/mnt/shared/ignition.done" ]]; do
        sleep 1 &
        wait
      done
      printf 'INFO %s ignition - Ignited. Starting ScyllaDB...\n' "$( date '+%Y-%m-%d %H:%M:%S,%3N' )" > /dev/stderr
      
      # TODO: This is where we should start ScyllaDB directly after the sidecar split #1942 
      exec /mnt/shared/scylla-operator sidecar \
      --feature-gates=AllAlpha=false,AllBeta=false,AutomaticTLSCertificates=true \
      --nodes-broadcast-address-type=ServiceClusterIP \
      --clients-broadcast-address-type=ServiceClusterIP \
      --service-name=$(SERVICE_NAME) \
      --cpu-count=$(CPU_COUNT) \
      --loglevel=2 \
       -- "$@"
      --
      
      --developer-mode=0
    State:          Running
      Started:      Tue, 28 Oct 2025 08:52:37 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  6Gi
    Requests:
      cpu:      1
      memory:   6Gi
...
Events:
  Type     Reason     Age                    From     Message
  ----     ------     ----                   ----     -------
  Warning  Unhealthy  3m36s (x575 over 93m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500

the pod log shows the following messag

E1028 12:54:25.234821       1 sidecar/controller.go:156] "Unhandled Error" err="syncing key 'scylla-cluster/testscylla-testscylla1-us-central1-b-0' failed: can't sync the HostID annotation: can't get host id to ip mapping: agent [HTTP 500] std::runtime_error (The gossiper is not ready yet)" logger="UnhandledError"

Do you have any recommendation what to look for to resolve the issue? For your information

  1. ingress and egress are completely open in the namespace
  2. the node has enough space for the requested limits for CPU and memory

I would appreciate it any help. I have been stuck with this for a while. Thanks in advance

Byungsoo

2

seems like you are trying to create a multi-DC cluster - as you mentioned 1st region and 2nd region?
if so the 2nd region needs to be able to access the seed host/IP from the 1st region to connect. do you have 2 k8s clusters?

my example here may help you. GitHub - tluck/Scylla-K8s-Example: Scylla K8s Example - End-to-End deployment.
let me know happy to get this sorted!

1 Like
3

Hello Thomas,

Thanks for your response

seems like you are trying to create a multi-DC cluster

No. The following is the copy of the example I refer to and there is only one DC. It creates statefulset per zone and they work as a single DC (single region).

kubectl apply --server-side -f=- <<EOF
apiVersion: scylla.scylladb.com/v1
kind: ScyllaCluster
metadata:
  name: scylladb
spec:
  repository: docker.io/scylladb/scylla
  version: 2025.1.5
  agentVersion: 3.5.1
  developerMode: false
  automaticOrphanedNodeCleanup: true
  sysctls:
  - fs.aio-max-nr=30000000
  datacenter:
    name: us-east-1
    racks:
    - name: us-east-1a
      members: 1
      scyllaConfig: scylladb-config
      storage:
        capacity: 100Gi
        storageClassName: scylladb-local-xfs
      resources:
        requests:
          cpu: 1
          memory: 8Gi
        limits:
          cpu: 1
          memory: 8Gi
      placement:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-east-1a 
              - key: scylla.scylladb.com/node-type
                operator: In
                values:
                - scylla
        tolerations:
        - key: scylla-operator.scylladb.com/dedicated
          operator: Equal
          value: scyllaclusters
          effect: NoSchedule
    - name: us-east-1b
      members: 1
      scyllaConfig: scylladb-config
      storage:
        capacity: 100Gi
        storageClassName: scylladb-local-xfs
      resources:
        requests:
          cpu: 1
          memory: 8Gi
        limits:
          cpu: 1
          memory: 8Gi
      placement:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-east-1b
              - key: scylla.scylladb.com/node-type
                operator: In
                values:
                - scylla
        tolerations:
        - key: scylla-operator.scylladb.com/dedicated
          operator: Equal
          value: scyllaclusters
          effect: NoSchedule
    - name: us-east-1c
      members: 1
      scyllaConfig: scylladb-config
      storage:
        capacity: 100Gi
        storageClassName: scylladb-local-xfs
      resources:
        requests:
          cpu: 1
          memory: 8Gi
        limits:
          cpu: 1
          memory: 8Gi
      placement:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-east-1c
              - key: scylla.scylladb.com/node-type
                operator: In
                values:
                - scylla
        tolerations:
        - key: scylla-operator.scylladb.com/dedicated
          operator: Equal
          value: scyllaclusters
          effect: NoSchedule
EOF

Thanks for sharing your repo. I will check it out and compare with what I did. Thanks again