Kubernetes doet not create correct directory

ScyllaDB
, , , ,
1

Installation details
kubernetes version: microk8s
#ScyllaDB version: 2025.1
#Cluster size: 3 nodes
os (RHEL/CentOS/Ubuntu/AWS AMI): Ubuntu

When I try to follow the gitops documentation for the scylla operator with a modified setup for the nodeconfig:

apiVersion: scylla{DOT}scylladb{DOT}com/v1alpha1
kind: NodeConfig
metadata:
name: scylladb-pool-1
spec:
localDiskSetup:
raids:
- name: nvmes
type: RAID0
RAID0:
devices:
nameRegex: ^/dev/sdb$
filesystems:
- device: /dev/md/nvmes
type: xfs
mounts:
- device: /dev/md/nvmes
mountPoint: /var/lib/persistent-volumes
unsupportedOptions:
- prjquota
placement:
nodeSelector:
scylla{DOT}scylladb{DOT}com/node-type: scylla
tolerations:
- effect: NoSchedule
key: scylla-operator{DOT}scylladb{DOT}com/dedicated
operator: Equal
value: scyllaclusters

When a pod tries to use the local csi driver (installed like the instructions in the gitops documentation). I get the following error in pod describe while the pod stays on init indefinitely:
MountVolume.SetUp failed for volume “pvc-9effa2df-4f8b-44b6-94a2-0697c2980a17” : rpc error: code = Internal desc = Failed to publish volume: can’t create target path at “/var/snap/microk8s/common/var/lib/kubelet/pods/ad3827af-2ee6-4913-9853-55205cd6f8c2/volumes/kubernetes{DOT}io~csi/pvc-9effa2df-4f8b-44b6-94a2-0697c2980a17/mount”: mkdir /var/snap/microk8s/common/var/lib/kubelet/pods/ad3827af-2ee6-4913-9853-55205cd6f8c2/volumes/kubernetes{DOT}io~csi/pvc-9effa2df-4f8b-44b6-94a2-0697c2980a17/mount: no such file or directory

All folders exist apart form the pvc-9effa2df-4f8b-44b6-94a2-0697c2980a17 one

When manually creating this folder it gets deleted when the pod tries to mount the volume again.

2

@mflendrich / @zimnx can you take a look ?

3

On MicroK8s the kubelet root lives under
/var/snap/microk8s/common/var/lib/kubelet, not /var/lib/kubelet.
In the upstream manifest the three hostPath volumes that the pods rely on still point to /var/lib/kubelet (see local-csi-driver/deploy/kubernetes/local-csi-driver/50_daemonset.yaml at master · scylladb/local-csi-driver · GitHub).
Because that directory is empty on a MicroK8s node, the driver container cannot see (or create)
/var/snap/…/kubelet/pods/<pod-uid>/volumes/kubernetes.io~csi/<pvc-uid>/mount and fails exactly as you observed.

There are two options to fix it, quick and dirty, on every node you could create a symlink
sudo ln -s /var/snap/microk8s/common/var/lib/kubelet /var/lib/kubelet, or patch the DaemonSet (or create your own overlay) so that all host paths reference the MicroK8s kubelet root.

2 Likes