Container runtime network not ready

[RogerSik]

Somehow my talos cluster is broken.

I have 1 control plane (which i can without problems scale up to 3 control plane) and 3 worker nodes.

The control planes says:

logs kubelet is spammed by:

$ talosctl logs -f kubelet    

talos-cp.hcloud.internal: {"ts":1737994182487.982,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994187489.6467,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994192491.8022,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994197492.8762,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994202494.1562,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994207496.0977,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}
talos-cp.hcloud.internal: {"ts":1737994212497.4988,"caller":"kubelet/kubelet.go:2902","msg":"Container runtime network not ready","networkReady":"NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"}

The node never goes ready because of:

Ready            False   Mon, 27 Jan 2025 17:09:16 +0100   Mon, 27 Jan 2025 14:31:19 +0100   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

I'm using the default flannel cni and i cant understand why this cluster is now broken and how to fix it. For any help I'm very appreciated. Is a homelab cluster running as a hobby.

My support zip:
support.zip

The control plane node is empty.

There are no pods in the kube-system they got removed after i restarted the control plane:

k get pod
No resources found in kube-system namespace.

because of that i stopped at the error message with NetworkPluginNotReady .

DaemonSet of flannel exist but dont get scheduled.

k get ds                                                                                              
NAME           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-flannel   4         0         0       0            0           <none>          25h
kube-proxy     4         0         0       0            0           <none>          25h
```

[smira]

Please attach talosctl support bundle.

Also you can a lot of yourself by doing kubectl get pods -A and looking into flannel pods - do they run or not, do they fail, etc.

[RogerSik]

Edited the post now with talosctl support bundle and more infos about kubectl

[RogerSik]

I created a separate fresh new talos cluster and there is networking working. The pods get scheduled and are visible. talosctl and kubectl says node is ready.

Strange.

[RogerSik]

Its working now. Sadly i can't say what the fix was :/ tried so many different things and did some break and after that node went healthy and everything begin to work again.

[DmitriyMV]

Don't be sad, software and hardware work in mysterious ways 😄 .

Jokes aside feel free to post this issue again if it ever comes back.