Originally from the User Slack
@Terence_Liu: Hi. We are trying to deploy the Scylla operator on a Kubernetes cluster (v1.23.8) through ArgoCD. Meeting this error:
this is the operator log:
I1121 18:48:56.874758 1 operator/operator.go:202] operator version "v1.14.0-rc.0-0-g95c2f90"
I1121 18:48:56.874779 1 flag/flags.go:64] FLAG: --burst="75"
I1121 18:48:56.874784 1 flag/flags.go:64] FLAG: --concurrent-syncs="50"
I1121 18:48:56.874786 1 flag/flags.go:64] FLAG: --cqls-ingress-port="0"
I1121 18:48:56.874788 1 flag/flags.go:64] FLAG: --crypto-key-buffer-delay="200ms"
I1121 18:48:56.874792 1 flag/flags.go:64] FLAG: --crypto-key-buffer-size-max="30"
I1121 18:48:56.874794 1 flag/flags.go:64] FLAG: --crypto-key-buffer-size-min="10"
I1121 18:48:56.874796 1 flag/flags.go:64] FLAG: --feature-gates=""
I1121 18:48:56.874803 1 flag/flags.go:64] FLAG: --help="false"
I1121 18:48:56.874806 1 flag/flags.go:64] FLAG: --image="scylladb/scylla-operator:1.14.0"
I1121 18:48:56.874809 1 flag/flags.go:64] FLAG: --kubeconfig=""
I1121 18:48:56.874811 1 flag/flags.go:64] FLAG: --leader-election-lease-duration="1m0s"
I1121 18:48:56.874813 1 flag/flags.go:64] FLAG: --leader-election-renew-deadline="35s"
I1121 18:48:56.874815 1 flag/flags.go:64] FLAG: --leader-election-retry-period="10s"
I1121 18:48:56.874817 1 flag/flags.go:64] FLAG: --loglevel="2"
I1121 18:48:56.874820 1 flag/flags.go:64] FLAG: --namespace="scylla-operator"
I1121 18:48:56.874821 1 flag/flags.go:64] FLAG: --qps="50"
I1121 18:48:56.874824 1 flag/flags.go:64] FLAG: --v="2"
I1121 18:48:56.874981 1 leaderelection/leaderelection.go:100] Starting leader election
I1121 18:48:56.874995 1 leaderelection/leaderelection.go:250] attempting to acquire leader lease scylla-operator/scylla-operator-lock...
I1121 18:50:03.394452 1 leaderelection/leaderelection.go:260] successfully acquired lease scylla-operator/scylla-operator-lock
I1121 18:50:03.395156 1 scylladbmonitoring/controller.go:564] "Starting controller" controller="ScyllaDBMonitoring"
I1121 18:50:03.395184 1 cache/shared_informer.go:311] Waiting for caches to sync for ScyllaDBMonitoringController
I1121 18:50:03.409386 1 scyllacluster/controller.go:279] "Starting controller" controller="ScyllaCluster"
I1121 18:50:03.409398 1 cache/shared_informer.go:311] Waiting for caches to sync for ScyllaClusterController
I1121 18:50:03.409414 1 orphanedpv/controller.go:167] "Starting controller" controller="OrphanedPV"
I1121 18:50:03.409417 1 cache/shared_informer.go:311] Waiting for caches to sync for OrphanedPVController
I1121 18:50:03.409427 1 nodeconfig/controller.go:467] "Starting controller" controller="NodeConfigController"
I1121 18:50:03.409432 1 cache/shared_informer.go:311] Waiting for caches to sync for NodeConfigController
I1121 18:50:03.409441 1 nodeconfigpod/controller.go:307] "Starting controller" controller="NodeConfigPodController"
I1121 18:50:03.409447 1 cache/shared_informer.go:311] Waiting for caches to sync for NodeConfigPodController
I1121 18:50:03.409456 1 scyllaoperatorconfig/controller.go:167] "Starting controller" controller="ScyllaOperatorConfigController"
I1121 18:50:03.409462 1 cache/shared_informer.go:311] Waiting for caches to sync for ScyllaOperatorConfigController
I1121 18:50:03.449732 1 cache/reflector.go:351] Caches populated for *v1alpha1.ScyllaOperatorConfig from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.451073 1 cache/reflector.go:351] Caches populated for *v1.ScyllaCluster from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.452991 1 cache/reflector.go:351] Caches populated for *v1.ServiceMonitor from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.464784 1 cache/reflector.go:351] Caches populated for *v1.Deployment from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.522934 1 cache/shared_informer.go:318] Caches are synced for ScyllaOperatorConfigController
I1121 18:50:03.523054 1 scyllaoperatorconfig/sync.go:21] "ScyllaOperatorConfig missing, creating a default one"
I1121 18:50:03.524723 1 cache/reflector.go:351] Caches populated for *v1.RoleBinding from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.526428 1 cache/reflector.go:351] Caches populated for *v1.DaemonSet from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.526785 1 cache/reflector.go:351] Caches populated for *v1.PrometheusRule from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.528412 1 cache/reflector.go:351] Caches populated for *v1.Prometheus from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.532159 1 cache/reflector.go:351] Caches populated for *v1alpha1.NodeConfig from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.532280 1 cache/reflector.go:351] Caches populated for *v1alpha1.ScyllaOperatorConfig from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.533108 1 cache/reflector.go:351] Caches populated for *v1.PersistentVolumeClaim from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.533715 1 cache/reflector.go:351] Caches populated for *v1.ServiceAccount from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.533974 1 cache/reflector.go:351] Caches populated for *v1alpha1.ScyllaDBMonitoring from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.535093 1 cache/reflector.go:351] Caches populated for *v1.ClusterRole from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.535290 1 cache/reflector.go:351] Caches populated for *v1.Namespace from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.535712 1 cache/reflector.go:351] Caches populated for *v1.ClusterRoleBinding from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.538070 1 cache/reflector.go:351] Caches populated for *v1.Role from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.543729 1 cache/reflector.go:351] Caches populated for *v1.Ingress from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.556532 1 cache/reflector.go:351] Caches populated for *v1.PersistentVolume from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.561175 1 cache/reflector.go:351] Caches populated for *v1.PodDisruptionBudget from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.569629 1 cache/reflector.go:351] Caches populated for *v1.Service from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.634695 1 cache/reflector.go:351] Caches populated for *v1.Job from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.640946 1 cache/reflector.go:351] Caches populated for *v1.ConfigMap from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.648636 1 cache/reflector.go:351] Caches populated for *v1.StatefulSet from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.653116 1 cache/reflector.go:351] Caches populated for *v1.Node from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
I1121 18:50:03.671679 1 cache/reflector.go:351] Caches populated for *v1.Secret from <http://k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229|k8s.io/client-go@v0.29.8/tools/cache/reflector.go:229>
E1121 18:50:03.692188 1 scyllaoperatorconfig/controller.go:151] syncing key 'cluster' failed: can't create scyllaoperatorconfig "cluster": Internal error occurred: failed calling webhook "<http://webhook.scylla.scylladb.com|webhook.scylla.scylladb.com>": failed to call webhook: Post "<https://scylla-operator-webhook.scylla-operator.svc:443/validate?timeout=10s>": x509: certificate is not valid for any names, but wanted to match scylla-operator-webhook.scylla-operator.svc
I1121 18:50:03.717545 1 scyllaoperatorconfig/sync.go:21] "ScyllaOperatorConfig missing, creating a default one"
E1121 18:50:03.756069 1 scyllaoperatorconfig/controller.go:151] syncing key 'cluster' failed: can't create scyllaoperatorconfig "cluster": Internal error occurred: failed calling webhook "<http://webhook.scylla.scylladb.com|webhook.scylla.scylladb.com>": failed to call webhook: Post "<https://scylla-operator-webhook.scylla-operator.svc:443/validate?timeout=10s>": x509: certificate is not valid for any names, but wanted to match scylla-operator-webhook.scylla-operator.svc
I1121 18:50:03.771295 1 scyllaoperatorconfig/sync.go:21] "ScyllaOperatorConfig missing, creating a default one"
E1121 18:50:03.838241 1 scyllaoperatorconfig/controller.go:151] syncing key 'cluster' failed: can't create scyllaoperatorconfig "cluster": Internal error occurred: failed calling webhook "<http://webhook.scylla.scylladb.com|webhook.scylla.scylladb.com>": failed to call webhook: Post "<https://scylla-operator-webhook.scylla-operator.svc:443/validate?timeout=10s>": x509: certificate is not valid for any names, but wanted to match scylla-operator-webhook.scylla-operator.svc
This is the webhook server log:
I1121 18:48:56.826216 1 operator/cmd.go:21] maxprocs: Leaving GOMAXPROCS=[16]: CPU quota undefined
I1121 18:48:56.826438 1 dynamiccertificates/dynamic_serving_content.go:113] "Loaded a new cert/key pair" name="serving-certs::/tmp/serving-certs/tls.crt::/tmp/serving-certs/tls.key"
I1121 18:48:56.826454 1 operator/webhooks.go:173] run-webhook-server version "v1.14.0-rc.0-0-g95c2f90"
I1121 18:48:56.826468 1 flag/flags.go:64] FLAG: --feature-gates=""
I1121 18:48:56.826478 1 flag/flags.go:64] FLAG: --help="false"
I1121 18:48:56.826484 1 flag/flags.go:64] FLAG: --insecure-generate-localhost-cert="false"
I1121 18:48:56.826488 1 flag/flags.go:64] FLAG: --loglevel="2"
I1121 18:48:56.826492 1 flag/flags.go:64] FLAG: --port="5000"
I1121 18:48:56.826501 1 flag/flags.go:64] FLAG: --tls-cert-file="/tmp/serving-certs/tls.crt"
I1121 18:48:56.826506 1 flag/flags.go:64] FLAG: --tls-private-key-file="/tmp/serving-certs/tls.key"
I1121 18:48:56.826510 1 flag/flags.go:64] FLAG: --v="2"
I1121 18:48:56.826756 1 dynamiccertificates/dynamic_serving_content.go:132] "Starting controller" name="serving-certs::/tmp/serving-certs/tls.crt::/tmp/serving-certs/tls.key"
I1121 18:48:56.826861 1 operator/webhooks.go:239] Starting HTTPS server on address "[::]:5000".
2024/11/21 18:50:04 http: TLS handshake error from 127.0.0.6:48227: EOF
2024/11/21 18:50:04 http: TLS handshake error from 127.0.0.6:41771: EOF
2024/11/21 18:50:04 http: TLS handshake error from 127.0.0.6:47157: EOF
2024/11/21 18:50:06 http: TLS handshake error from 127.0.0.6:60505: EOF
2024/11/21 18:50:45 http: TLS handshake error from 127.0.0.6:52633: EOF
2024/11/21 18:51:26 http: TLS handshake error from 127.0.0.6:44627: EOF
2024/11/21 18:55:32 http: TLS handshake error from 127.0.0.6:36593: EOF
I know our k8s cluster version is two years late. We had success deploying with cert on a recently versioned k8s. Does the operator only support later versions?
@Tomas_Nozicka: please file an issue on GH and provide must-gather so someone can look but this doesn’t seem related to kube version https://operator.docs.scylladb.com/v1.14/releases.html#support-matrix
Releases | ScyllaDB Docs
but this seems like an issue with cert-manager probably
@Terence_Liu: @Bradley_Stock
@Bradley_Stock: After playing with it for a while I can confirm that this is related to Istio. I’ve been able to get everything working as expected when I disable the Istio sidecar
I worked around the issue by adding the following annotations to the webhook-server deployment:
spec:
template:
metadata:
annotations:
<http://traffic.sidecar.istio.io/excludeInboundPorts|traffic.sidecar.istio.io/excludeInboundPorts>: "443,5000"
<http://traffic.sidecar.istio.io/excludeOutboundPorts|traffic.sidecar.istio.io/excludeOutboundPorts>: "443,5000"
This essentially disables istio for those specific ports for the webhook-server.
@Guy: Hey @Terence_Liu and @Bradley_Stock, following up on this, did you file an issue?
@Terence_Liu: I’ll let @Bradley_Stock tell more on Monday but I think it was due to conflict with istio’s TLS settings - in that case it seems to be a user config issue