Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

[huanghaiqing1]

Setup a ha based k8s and deploy first control plane node with calico, met below warning.

tigera-operator.yaml + custom-resources.yaml are deployed with no error. But IPv4 BGP status reports nothing. calico-node-qcv5g runs but kubectl describe with warning like below shows. And I have no OS level acl/iptables/nft block about any rules. All my environment is based on vmware workstation. Any prerequisite requir3ed about calico?

kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

[root@k8sma manifests]# calicoctl node status
Calico process is running.

IPv4 BGP status
No IPv4 peers found.

IPv6 BGP status
No IPv6 peers found.

[root@k8sma manifests]# kubectl get pods --all-namespaces 
NAMESPACE          NAME                                       READY   STATUS    RESTARTS       AGE
calico-apiserver   calico-apiserver-76859b547-nj6gh           1/1     Running   1 (22m ago)    41h
calico-apiserver   calico-apiserver-76859b547-v9rll           1/1     Running   1 (22m ago)    41h
calico-system      calico-kube-controllers-6577875f76-cbhzj   1/1     Running   1 (22m ago)    41h
calico-system      calico-node-qcv5g                          1/1     Running   2 (22m ago)    41h
calico-system      calico-typha-7b7497cbd7-c86g8              1/1     Running   2 (22m ago)    41h
calico-system      csi-node-driver-n57gh                      2/2     Running   2 (22m ago)    41h
kube-system        coredns-cb4864fb5-nxp42                    1/1     Running   1 (22m ago)    41h
kube-system        coredns-cb4864fb5-pkb2v                    1/1     Running   1 (22m ago)    41h
kube-system        etcd-k8sma                                 1/1     Running   50 (22m ago)   41h
kube-system        kube-apiserver-k8sma                       1/1     Running   10 (22m ago)   41h
kube-system        kube-controller-manager-k8sma              1/1     Running   10 (22m ago)   41h
kube-system        kube-proxy-4nwv2                           1/1     Running   2 (22m ago)    41h
kube-system        kube-scheduler-k8sma                       1/1     Running   10 (22m ago)   41h
tigera-operator    tigera-operator-7f57f79bb6-xjkmx           1/1     Running   2 (22m ago)    41h

[root@k8sma manifests]# kubectl describe pod calico-node-qcv5g -n calico-system 
Name:                 calico-node-qcv5g
Namespace:            calico-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Service Account:      calico-node
Node:                 k8sma/192.168.31.111
Start Time:           Fri, 20 Dec 2024 17:19:43 +0800
Labels:               app.kubernetes.io/name=calico-node
                      controller-revision-hash=68b59d4c48
                      k8s-app=calico-node
                      pod-template-generation=2
Annotations:          hash.operator.tigera.io/cni-config: 59f12efe409df4d285e1cf584d1738c376199c6b
                      hash.operator.tigera.io/system: fdde45054a8ae4f629960ce37570929502e59449
                      tigera-operator.hash.operator.tigera.io/tigera-ca-private: d98fed76648a5635d3d7eca6518cbef0ba4a9f42
Status:               Running
IP:                   192.168.31.111
IPs:
  IP:           192.168.31.111
Controlled By:  DaemonSet/calico-node
Init Containers:
  flexvol-driver:
    Container ID:    containerd://850f4eb7a903a851175d7a6e9f1628ecd6f32d610475665b9ec2adc7e808b14c
    Image:           docker.io/calico/pod2daemon-flexvol:v3.29.1
    Image ID:        docker.io/calico/pod2daemon-flexvol@sha256:b39582ca0014605e8887a70e2f2fb103c8149d28d1aa6482fb5589e3c7c06197
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    State:           Terminated
      Reason:        Completed
      Exit Code:     0
      Started:       Sun, 22 Dec 2024 10:28:49 +0800
      Finished:      Sun, 22 Dec 2024 10:28:49 +0800
    Ready:           True
    Restart Count:   2
    Environment:     <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fllq8 (ro)
  install-cni:
    Container ID:    containerd://e35b9985b615f92960ce71ca741f3b3befd967f592cf9214d5bea56c0813d698
    Image:           docker.io/calico/cni:v3.29.1
    Image ID:        docker.io/calico/cni@sha256:fb35441ca9be59ebc2a3a00e6951f879d0edbecb3c1314defa3c4e927f4628c1
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    Command:
      /opt/cni/bin/install
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 22 Dec 2024 10:28:50 +0800
      Finished:     Sun, 22 Dec 2024 10:28:50 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:            10-calico.conflist
      SLEEP:                    false
      CNI_NET_DIR:              /etc/cni/net.d
      CNI_NETWORK_CONFIG:       <set to the key 'config' of config map 'cni-config'>  Optional: false
      KUBERNETES_SERVICE_HOST:  192.1.0.1
      KUBERNETES_SERVICE_PORT:  443
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fllq8 (ro)
Containers:
  calico-node:
    Container ID:    containerd://90452e86421476f8bd896c5a0325a0eca66675014953f93a6bda2f9e4c38cb47
    Image:           docker.io/calico/node:v3.29.1
    Image ID:        docker.io/calico/node@sha256:a819b2cb3dbeaa3849e99f40a5af60641bedc4d063456a718ad089ef38250389
    Port:            <none>
    Host Port:       <none>
    SeccompProfile:  RuntimeDefault
    State:           Running
      Started:       Sun, 22 Dec 2024 10:28:51 +0800
    Last State:      Terminated
      Reason:        Unknown
      Exit Code:     255
      Started:       Sat, 21 Dec 2024 12:57:51 +0800
      Finished:      Sun, 22 Dec 2024 10:28:10 +0800
    Ready:           True
    Restart Count:   2
    Liveness:        http-get http://localhost:9099/liveness delay=0s timeout=10s period=60s #success=1 #failure=3
    Readiness:       exec [/bin/calico-node -bird-ready -felix-ready] delay=0s timeout=10s period=30s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      CLUSTER_TYPE:                       k8s,operator,bgp
      CALICO_DISABLE_FILE_LOGGING:        false
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_HEALTHENABLED:                true
      FELIX_HEALTHPORT:                   9099
      NODENAME:                            (v1:spec.nodeName)
      NAMESPACE:                          calico-system (v1:metadata.namespace)
      FELIX_TYPHAK8SNAMESPACE:            calico-system
      FELIX_TYPHAK8SSERVICENAME:          calico-typha
      FELIX_TYPHACAFILE:                  /etc/pki/tls/certs/tigera-ca-bundle.crt
      FELIX_TYPHACERTFILE:                /node-certs/tls.crt
      FELIX_TYPHAKEYFILE:                 /node-certs/tls.key
      FIPS_MODE_ENABLED:                  false
      NO_DEFAULT_POOLS:                   true
      FELIX_TYPHACN:                      typha-server
      CALICO_MANAGE_CNI:                  true
      CALICO_NETWORKING_BACKEND:          bird
      IP:                                 autodetect
      IP_AUTODETECTION_METHOD:            first-found
      IP6:                                none
      FELIX_IPV6SUPPORT:                  false
      KUBERNETES_SERVICE_HOST:            192.1.0.1
      KUBERNETES_SERVICE_PORT:            443
    Mounts:
      /etc/pki/tls/cert.pem from tigera-ca-bundle (ro,path="ca-bundle.crt")
      /etc/pki/tls/certs from tigera-ca-bundle (ro)
      /host/etc/cni/net.d from cni-net-dir (rw)
      /lib/modules from lib-modules (ro)
      /node-certs from node-certs (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/log/calico/cni from cni-log-dir (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fllq8 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  tigera-ca-bundle:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      tigera-ca-bundle
    Optional:  false
  node-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  node-certs
    Optional:    false
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  DirectoryOrCreate
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  DirectoryOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  DirectoryOrCreate
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  cni-log-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log/calico/cni
    HostPathType:  
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  kube-api-access-fllq8:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 :NoSchedule op=Exists
                             :NoExecute op=Exists
                             CriticalAddonsOnly op=Exists
                             node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason          Age                From     Message
  ----     ------          ----               ----     -------
  Normal   SandboxChanged  24m                kubelet  Pod sandbox changed, it will be killed and re-created.
  Normal   Pulled          24m                kubelet  Container image "docker.io/calico/pod2daemon-flexvol:v3.29.1" already present on machine
  Normal   Created         24m                kubelet  Created container flexvol-driver
  Normal   Started         24m                kubelet  Started container flexvol-driver
  Normal   Pulled          24m                kubelet  Container image "docker.io/calico/cni:v3.29.1" already present on machine
  Normal   Created         24m                kubelet  Created container install-cni
  Normal   Started         24m                kubelet  Started container install-cni
  Normal   Pulled          24m                kubelet  Container image "docker.io/calico/node:v3.29.1" already present on machine
  Normal   Created         24m                kubelet  Created container calico-node
  Normal   Started         24m                kubelet  Started container calico-node
  Warning  Unhealthy       24m                kubelet  Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/bird/bird.ctl: connect: no such file or directory
  Warning  Unhealthy       23m (x3 over 24m)  kubelet  Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

[huangzeqi]

calico.zip
This is my local tigera-operator.yaml + custom-resources.yaml. I only adjusted download images with my private registry hub address and my pod network segment.

[caseydavenport]

@huanghaiqing1 @huangzeqi probably a good next step is to look at the calico-node logs themselves to see if there are any errors.

This message - dial unix /var/run/bird/bird.ctl: connect: no such file or directory typically means that confd is encountering issues generating the BIRD configuration files, and as such BIRD is failing to start. Those errors would show up in the calico/node logs.

[mazdakn]

@huanghaiqing1 @huangzeqi any update on this issue? Have you managed to find the root cause?

[mohammadt-github]

I have this issue to. do you have any resolution?? @caseydavenport @huangzeqi @huanghaiqing1
After install calico 3.29.1 in my k8s 1.32.1 I have this log : "dial unix /var/run/bird/bird.ctl: connect: no such file or directory"
after 60 second calico pod restart and this log generate in k describe pods calico-node-4fncc -n kube-system
"kubelet Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused"

[tomastigera]

@mohammadt-github ⬆

probably a good next step is to look at the calico-node logs themselves to see if there are any errors.

[mohammadt-github]

@tomastigera
Thanks. I saw that, noting is weird.
Is this log is related to my problem?

tunnel-ip-allocator/allocateip.go 340: Current address is still valid, do nothing currentAddr="10.244.34.192" type="ipipTunnelAddress"
Calico node started successfully
bird: Unable to open configuration file /etc/calico/confd/config/bird6.cfg: No such file or directory
bird: Unable to open configuration file /etc/calico/confd/config/bird.cfg: No such file or directory
2025-01-28 21:03:32.611 [WARNING][109] cni-config-monitor/winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2025-01-28 21:03:32.615 [INFO][108] monitor-addresses/startup.go 439: Early log level set to info
2025-01-28 21:03:32.617 [INFO][104] status-reporter/startup.go 439: Early log level set to info
2025-01-28 21:03:32.617 [INFO][105] confd/config.go 83: Skipping confd config file.
2025-01-28 21:03:32.617 [INFO][105] confd/run.go 18: Starting calico-confd
------------

2025-01-28 21:03:32.850 [INFO][110] felix/connecttime.go 85: Failed to list BPF programs.  Assuming not supported/nothing to clean up. error=exit status 255 output="{\n    \"error\": \"can't query bpf programs attached to /run/calico/cgroup: No such device\"\n}\n"
2025-01-28 21:03:32.850 [INFO][110] felix/int_dataplane.go 759: Failed to remove BPF connect-time load balancer, ignoring. error=exit status 255

this is output of checksystem

[mohammadt-github]

@huanghaiqing1 how resolve this problem.? could you please explain solution if resolved that ?