CephFS Mount Failed Kernel

[suse-coder]

Hi,
i use csi for RBD and CephFS to provision volumes (PVC) via storageclasses from my external ceph cluster (over the client api).

RBD works well and the pod and PVC is created, but for cephfs i get a talos kernel error that the module ceph does not exist (modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos) . How can I solve that:

Namespace:        default
Priority:         0
Service Account:  default
Node:             talos-p7f-wro/192.168.178.78
Start Time:       Sat, 22 Feb 2025 10:12:55 +0000
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Containers:
  test-container:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      echo "Test content for CephFS" > /mnt/cephfs/test.txt
      echo "CephFS file created. Sleeping..."
      sleep 3600
      
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /mnt/cephfs from cephfs-storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xc8v7 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   False 
  Initialized                 True 
  Ready                       False 
  ContainersReady             False 
  PodScheduled                True 
Volumes:
  cephfs-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  cephfs-test-pvc
    ReadOnly:   false
  kube-api-access-xc8v7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  118s  default-scheduler  0/4 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.
  Normal   Scheduled         117s  default-scheduler  Successfully assigned default/cephfs-test-pod to talos-p7f-wro
  Warning  FailedMount       117s  kubelet            MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-3562981921,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:12:56.138+0000 7f824977a000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  117s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-4234836738,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:12:56.722+0000 7f1558c6f000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  116s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-3175506078,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:12:57.822+0000 7fc4269b0000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  114s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-4238067649,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:12:59.939+0000 7f6e5c06e000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  109s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-390123845,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:13:04.079+0000 7f9e6de5a000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  101s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-4287281797,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:13:12.151+0000 7f9034011000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  85s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-3898604505,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:13:28.291+0000 7f99c6f5b000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied
  Warning  FailedMount  53s  kubelet  MountVolume.MountDevice failed for volume "pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714" : rpc error: code = Internal desc = an error (exit status 32) occurred while running mount args: [-t ceph 192.168.178.88:6789:/volumes/csi/csi-vol-f0f4ebc7-fdb9-456d-ac11-e2bc3f4a2438/19ece4f4-1398-45ec-a767-672cf9813960 /var/lib/kubelet/plugins/kubernetes.io/csi/cephfs.csi.ceph.com/1c8cfcce70139611de9090740366c46e196775cbf4b8f34c416462a68c8d0444/globalmount -o name=services-fs,secretfile=/tmp/csi/keys/keyfile-1510863468,mds_namespace=cephfs-services,_netdev] stderr: unable to get monitor info from DNS SRV with service name: ceph-mon
2025-02-22T10:14:00.367+0000 7fafbad03000 -1 failed for service _ceph-mon._tcp
modprobe: FATAL: Module ceph not found in directory /lib/modules/6.12.5-talos
mount error 13 = Permission denied

Name:          cephfs-test-pvc
Namespace:     default
StorageClass:  services-filesystem
Status:        Bound
Volume:        pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWX
VolumeMode:    Filesystem
Used By:       cephfs-test-pod
Events:
  Type    Reason                 Age                  From                                                                                                   Message
  ----    ------                 ----                 ----                                                                                                   -------
  Normal  Provisioning           5m4s                 cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5cdb9dd5c4-r8966_d8a93eb2-9f75-40c0-a5b4-de5fa0693d21  External provisioner is provisioning volume for claim "default/cephfs-test-pvc"
  Normal  ExternalProvisioning   5m4s (x2 over 5m4s)  persistentvolume-controller                                                                            Waiting for a volume to be created either by the external provisioner 'cephfs.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  ProvisioningSucceeded  5m4s                 cephfs.csi.ceph.com_ceph-csi-cephfs-provisioner-5cdb9dd5c4-r8966_d8a93eb2-9f75-40c0-a5b4-de5fa0693d21  Successfully provisioned volume pvc-72ffd8c8-9f23-41b2-ab48-90e5e420a714

timoni csi bundles:

bundle: {
	apiVersion: "v1alpha1"
	name:       "ceph-csi-bundle"
	instances: {
		"ceph-csi-rbd": {
			module: {
				url:     "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
				version: "latest"
			}
			namespace: "ceph"
			values: {
				repository: {
					url: "https://ceph.github.io/csi-charts"
				}
				chart: {
					name:    "ceph-csi-rbd"
					version: "*"
				}
				helmValues: {
					configMapName: "ceph-csi-config"
					selinuxMount: false  
					provisioner: {
						enabled: true
					}
					nodeplugin: {
						enabled: true
					}
				}
			}
		},
		"ceph-csi-cephfs": {
			module: {
				url:     "oci://ghcr.io/stefanprodan/modules/flux-helm-release"
				version: "latest"
			}
			namespace: "ceph"
			values: {
				repository: {
					url: "https://ceph.github.io/csi-charts"
				}
				chart: {
					name:    "ceph-csi-cephfs"
					version: "*"
				}
				helmValues: {
					configMapName: "ceph-csi-config"
					selinuxMount: false  
					provisioner: {
						enabled: true
					}
					nodeplugin: {
						enabled: true
					}
				}
			}
		}
	}
}

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: services-block
  namespace: ceph
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: "dbb33f93-ec5d-4232-a26c-471f71d91ad3"
  pool: "rbdservices"    
  imageFeatures: "layering"
  csi.storage.k8s.io/provisioner-secret-name: "services-rbd-secret"
  csi.storage.k8s.io/provisioner-secret-namespace: "ceph"
  csi.storage.k8s.io/controller-expand-secret-name: "services-rbd-secret"
  csi.storage.k8s.io/controller-expand-secret-namespace: "ceph"
  csi.storage.k8s.io/node-stage-secret-name: "services-rbd-secret"
  csi.storage.k8s.io/node-stage-secret-namespace: "ceph"
  csi.storage.k8s.io/fstype: "ext4"
reclaimPolicy: Retain
allowVolumeExpansion: true
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: services-filesystem
  namespace: ceph
provisioner: cephfs.csi.ceph.com
parameters:
  clusterID: "dbb33f93-ec5d-4232-a26c-471f71d91ad3"
  fsName: "cephfs-services"   
  pool: "cephfs-services"      
  csi.storage.k8s.io/provisioner-secret-name: "services-fs-secret"
  csi.storage.k8s.io/provisioner-secret-namespace: "ceph"
  csi.storage.k8s.io/controller-expand-secret-name: "services-fs-secret"
  csi.storage.k8s.io/controller-expand-secret-namespace: "ceph"
  csi.storage.k8s.io/node-stage-secret-name: "services-fs-secret"
  csi.storage.k8s.io/node-stage-secret-namespace: "ceph"
reclaimPolicy: Retain
allowVolumeExpansion: true

[suse-coder]

Fixed it. Was a Ceph error. I used a subvolume group /csi and that needed to be removed:
correct:
sudo microceph.ceph auth get-or-create client.services-fs
mon 'allow r'
mds 'allow rw'
osd 'allow rw pool=cephfs-services, allow rw pool=cephfs-services-metadata'
mgr 'allow r'

wrong:
sudo microceph.ceph auth get-or-create client.services-fs
mon 'allow r'
mds 'allow rw path=/csi'
osd 'allow rw pool=cephfs-services, allow rw pool=cephfs-services-metadata'
mgr 'allow rw'