Running with Operator on GKE, cluster won't start with the Guaranteed QoS class

Guy · September 24, 2024, 9:16am

@Łukasz_Sanokowski: I have a problem with Scylla spinned up via Operator in latest version 1.13.0 on GKE. It won’t start with Guaranteed QoS class, while it works on exactly the same stack, but on Burstable class.

The log of scylla pod running in Guranteed class is following (thats whole, it is getting stuck there):

 Kubectl logs scylla-europe-west4-europe-west4-a-0 scylla
I0909 10:49:20.511209       1 operator/cmd.go:21] maxprocs: Updating GOMAXPROCS=[6]: determined from CPU quota
I0909 10:49:20.522214       1 operator/sidecar.go:167] sidecar version "v1.13.0-rc.0-2-g7f37771"
I0909 10:49:20.522240       1 flag/flags.go:64] FLAG: --burst="5"
I0909 10:49:20.522248       1 flag/flags.go:64] FLAG: --clients-broadcast-address-type="ServiceClusterIP"
I0909 10:49:20.522254       1 flag/flags.go:64] FLAG: --cpu-count="6"
I0909 10:49:20.522258       1 flag/flags.go:64] FLAG: --external-seeds="[]"
I0909 10:49:20.522272       1 flag/flags.go:64] FLAG: --feature-gates="AllAlpha=false,AllBeta=false,AutomaticTLSCertificates=true"
I0909 10:49:20.522289       1 flag/flags.go:64] FLAG: --help="false"
I0909 10:49:20.522295       1 flag/flags.go:64] FLAG: --kubeconfig=""
I0909 10:49:20.522300       1 flag/flags.go:64] FLAG: --loglevel="2"
I0909 10:49:20.522305       1 flag/flags.go:64] FLAG: --namespace="scylla"
I0909 10:49:20.522310       1 flag/flags.go:64] FLAG: --nodes-broadcast-address-type="ServiceClusterIP"
I0909 10:49:20.522315       1 flag/flags.go:64] FLAG: --qps="2"
I0909 10:49:20.522327       1 flag/flags.go:64] FLAG: --service-name="scylla-europe-west4-europe-west4-a-0"
I0909 10:49:20.522332       1 flag/flags.go:64] FLAG: --v="2"
I0909 10:49:20.525843       1 operator/sidecar.go:207] "Waiting for single service informer caches to sync"
I0909 10:49:20.537786       1 cache/reflector.go:351] Caches populated for *v1.Service from k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229
I0909 10:49:20.626652       1 operator/sidecar.go:213] "Waiting for Service availability and IP address" Service="scylla/scylla-europe-west4-europe-west4-a-0"
I0909 10:49:20.631351       1 cache/reflector.go:351] Caches populated for *v1.Service from k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229
I0909 10:49:20.632930       1 operator/sidecar.go:236] "Waiting for Pod to have IP address assigned and scylla ContainerID set" Pod="scylla/scylla-europe-west4-europe-west4-a-0"
I0909 10:49:20.640316       1 cache/reflector.go:351] Caches populated for *v1.Pod from k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229
E0909 10:49:20.641920       1 cache/reflector.go:147] k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229: Failed to watch *v1.Pod: unknown (get pods)
I0909 10:49:21.540191       1 cache/reflector.go:351] Caches populated for *v1.Pod from k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229
E0909 10:49:21.541665       1 cache/reflector.go:147] k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229: Failed to watch *v1.Pod: unknown (get pods)
I0909 10:49:21.541729       1 operator/sidecar.go:286] "Waiting for NodeConfig's data ConfigMap " Selector="scylla-operator.scylladb.com/config-map-type=NodeConfigData,scylla-operator.scylladb.com/owner-uid=5f42f629-d691-411e-922f-f97ecceabf47"
I0909 10:49:21.547567       1 cache/reflector.go:351] Caches populated for *v1.ConfigMap from k8s.io/client-go@v0.29.5/tools/cache/reflector.go:229

After the simple change of disabling guaranteed class by commenting out:

    agentResources:
      limits:
        cpu: 1
        memory: 1Gi

exactly the same setup works, scylla pod is loading successfully as the other containers are.

@Maciej_Zimnoch: Is your kubelet cpuManagerPolicy set to static?

@Łukasz_Sanokowski: You are right @Maciej_Zimnoch, setting it to static did the job, thank you