cannot get multi-home configuration to work

[dougcooper]

I'm evaluating calico and i need to configure it to manage the data plane on a specific interface. I'm currently configured using BIRD with no overlay. Ive configured IP_AUTODETECTION_METHOD set to interface=<interface name>, but it always chooses the first available interface and not the one i specified. I validate using curl inside the cluster and running tcpdump on the interface.

[tomastigera]

Do you install from manifest or using operator? If you use operator, then changing env vars wouldn't work

[dougcooper]

there isnt anything interesting in the log. i set FELIX_LOGSEVERITYSCREEN to warning. please let me know if you have better suggested settings of increasing log verbosity. kube-system-calico-node-4547x-1738952665162675086.log

[tomastigera]

I'd set it to debug

[dougcooper]

Alot more interesting data in this log kube-system-calico-node-pkq6t-1739217340652963645.log. I have an identical two-node system with two interfaces each:

Node 1

1. enp1s0 - 192.168.122.2
2. enp7s0 - 192.168.3.2

Node 2

1. enp1s0 - 192.168.122.3
2. enp7s0 - 192.168.3.3

In the attached log the curl pod was scheduled on node 2. Im expecting curl to send data on enp7s0 but it comes out on enp1s0. If there is any other config info i can provide please let me know.

[dougcooper]

i just noticed i have natOutgoing set to true and im not testing within cluster traffic. when i run pods communicating with each other on different nodes it is on the expected interface. so it seems snat it defaulting to the first interface and not the one pods are using for internal traffic.

[tomastigera]

It really depends on how you setup routing to the external world. nat-outgoing will use the IP of the interface that is sending the data out

[dougcooper]

im using kubespray which installs from manifest.

[tomastigera]

what version of calico do you use?

[dougcooper]

v3.28.1