calico should properly route overlay traffic between regular and host networked pods

[igcherkaev]

Expected Behavior

Traffic initiated by regular pods to in-cluster services backed by pods using host network (e.g. kube-apiserver) is allowed and flowing.

Current Behavior

Traffic initiated by regular pods to in-cluster services backed by pods using host network (e.g. kube-apiserver) is dropped by wireguard allowed IPs policies set by calico for each peer on wireguard.cali interface.

Possible Solution

Possible solution is to allow users adjust the allowed-ips for peers on the wireguard interface.

  wireguardEnabled: true
  wireguardAllowedIPs: '10.142.53.31/32,10.142.53.32/32'

An optional field for FelixConfiguration (or maybe IPPool?), which would be concatenated with the list of computed CIDR blocks that calico configures peers with on the wireguard interface.

Steps to Reproduce (for bugs)

Note: @frozenprocess was able to reproduce this issue.

1. Enable wireguard debug on kubernetes nodes. (echo module wireguard +p > /sys/kernel/debug/dynamic_debug/control)
2. Run kube-apiserver pods or any other pods using host network.
3. Run regular pods and try to contact the host-network pods.
4. On the node with the regular pod you'll notice errors in dmesg similar to:

[13590.173457] wireguard: wireguard.cali: Packet has unallowed src IP (10.142.53.61) from peer 12 (10.142.53.61:51820)
[13590.182499] wireguard: wireguard.cali: Packet has unallowed src IP (10.142.53.32) from peer 15 (10.142.53.32:51820)
[13590.230851] wireguard: wireguard.cali: Packet has unallowed src IP (10.142.53.35) from peer 13 (10.142.53.35:51820)

Context

Once wireguard is enabled, no pods can talk to in-cluster kubernetes api service.

Your Environment

• Calico version: 3.29.1
• Calico dataplane (iptables, windows etc.): ebpf, linux
• Orchestrator version (e.g. kubernetes, mesos, rkt): Kubernetes 1.29.12
• Operating System and version: Flatcar Container Linux by Kinvolk 4081.2.1 (Oklo) 6.6.65-flatcar containerd://1.7.21

Felix Configuration:

apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
  name: default
spec:
  bpfEnabled: true
  bpfExternalServiceMode: DSR
  bpfKubeProxyIptablesCleanupEnabled: true
  bpfLogLevel: ""
  logSeverityScreen: Info
  reportingInterval: 0s
  ipipEnabled: false
  vxlanEnabled: false
  wireguardEnabled: true
  bpfEnforceRPF: Disabled

IPPools configuration:

kind: IPPoolList
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
  kind: IPPool
  metadata:
    name: 10-161-128-0-17
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: 10.161.128.0/17
    ipipMode: Never
    nodeSelector: all()
    natOutgoing: false

BGP is in use, DSR is on, vxlan is likely a no-go option.

[tomastigera]

Accoring to @frozenprocess it is not ebpf related/only issue.

[tomastigera]

The only way to make it work atm is to have natOutgoing set on your ippools, Note that it would make any connections to/from host networked pods bypass wireguard.

[igcherkaev]

The only way to make it work atm is to have natOutgoing set on your ippools, Note that it would make any connections to/from host networked pods bypass wireguard.

Makes sense, but we can wait until a better solution is available. Adding an extra IP range to the wg allowed ips would be good enough for us atm. But if there's a better way to make it work, we have no objections :)

[igcherkaev]

To me it's either let users adjust allowed ips on wireguard peers, or let calico automatically do that, or somehow implement somewhat of PBR (policy based routing) so that the traffic from host-network pods goes back the way it came and not via the wg iface per the routing table. Just a thought.

[tomastigera]

Allowing some extra IPs on the wg device would only paper over the real issue similar to how natoutgoing does it. Routing is the real fix and that we surely want to look into.