When using Calico CNI, the MTU of the pod NIC is set to 1500 with a random probability.

[iksoon-park]

There is a problem with the MTU settings of the calico veth NIC (ex : calif4fad8d7ab0).

The calico specs are as follows:

• datastore is etcd and calico is deployed on kubernetes
  • kubernetes : v1.28.3
• calico/node version : 3.28.2
• veth_mtu: "0"
• FELIX_MTUIFACEPATTERN : default ^((en|wl|ww|sl|ib)[Pcopsvx].|(eth|wlan|wwan).)
• Calico network mode : VXLAN

Test host NIC environment is as follows:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP group default qlen 1000
    link/ether fa:16:3e:34:5a:21 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 192.168.0.88/24 metric 100 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 78791sec preferred_lft 78791sec
    inet6 fe80::f816:3eff:fe34:5a21/64 scope link
       valid_lft forever preferred_lft forever
4: calif4fad8d7ab0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-bc7d007a-6242-6915-c40b-f1560f749e92
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
5: cali4766628d1a3@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-c1348e0c-5e56-c852-a764-a2433460eaff
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever
6: calif89f0257d27@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
    link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netns cni-65e55f68-24b7-6372-6eaa-8e6a1a6d0340
    inet6 fe80::ecee:eeff:feee:eeee/64 scope link
       valid_lft forever preferred_lft forever

Also found the following in felix log
felix/int_dataplane.go 1086: Determined pod MTU mtu=1400

The MTU of the veth NIC assigned to the Pod should be calculated as follows and set to 1400

• eth0 interface MTU (1450) - VXLAN header (50) = 1400

However, with some probability, 1500 is allocated to a specific veth0 interface, such as the calif4fad8d7ab0 interface.

Expected Behavior

All calico veth interfaces must have their MTU set to 1400

Current Behavior

There is a calico veth NIC that is randomly set to 1500.

Possible Solution

Pod delete and deploy or restart nodes

Steps to Reproduce (for bugs)

1. kubernetes cluster create
2. Calico CNI deploy
3. pod Deploy
4. Check worker node pod veth NIC MTU

Context

k8s cluster networking error

Your Environment

• Calico version
  • calico/node:v3.24.1, v3.28.0, v3.28.2
  • calico/typha:v3.24.1, v3.28.0, v3.28.2
  • calico/kube-controllers:v3.24.1, v3.28.0, v3.28.2
• Calico dataplane (iptables, windows etc.)
  • Calico VXLAN mode
  • Calico eBPF true and Networking Direct
• Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes v1.24.3, v1.26.3, v1.27.3, v1.28.3, v1.29.3
• Operating System and version: ubuntu 22.04, ubuntu 20.04, rocky 8.9
• Link to your project (optional):

[iksoon-park]

@tomastigera

Here’s a more precise description of how to reproduce the issue:

• Remove calico-node daemonset(Felix)
• Remove /var/lib/calico/mtu file on worker node
• deployment of test pods on worker node
  • Since there is no CNI, it is not deployed and is waiting in ContainerCreating state
• Deploy calico-node daemonset(Felix)
• Check the status of pods that were mass deployed - some have MTU set to 1500

Here are the expected related codes for the issue.

In the case where the problem occurred, the error below should have occurred in the MTUFromFile function, but it is not confirmed in the node or felix log.

logrus.Infof("File %s does not exist", filename)

The error check only checks whether the file exists,
but the internal MTU value does not exist, so the problem is expected to have occurred.

[tomastigera]

I guess the reason for that is that some pods are created after calico-node (Felix part) starts up and does all the detection while some are started right after CNI is available and the file is not written yet. Idk why the log is not printed though if there is no file yet 🤔

[iksoon-park]

@tomastigera
Currently only validating the existence of the file, but I think a logic to verify whether the file contains content should also be added.