How to tell KubeSpan to connect over a particular network?

[dhess]

Hi, just want to say that Talos is truly impressive tech!

I have a couple of Hetzner dedicated hosts running Proxmox, each connected via a Hetzner vSwitch. The vSwitch has a public IPv4 /29 subnet assigned to it, and I've assigned the public IPv4's from this subnet to a few Proxmox VMs, each running Talos. I've also configured another VLAN on the vSwitch for private traffic (e.g., over RFC 1918 networks, and an IPv6 ULA).

I've managed to get the Talos VMs configured so that etcd, kubelet, etc. all advertise/listen over the private traffic VLAN using RFC 1918 and ULA IPs, but I can't seem to convince KubeSpan to do so. talosctl get kubespanpeerstatuses shows that all KubeSpan endpoints have been chosen from the public IPv4 subnet. I'd prefer not to use the public IPv4 subnet for KubeSpan traffic, as Hetzner charge extra for traffic that exceeds 1 TB/mo on the public IPv4 subnet, which I believe the inter-node KubeSpan traffic would qualify as, even if it's technically within Hetzner's network. (I'm not 100% certain on this, however.)

I probably I don't really need KubeSpan in this case, because for the most part I can trust the Hetzner vSwitch, but I'd like to have the option anyway. How can I configure KubeSpan to connect via a particular subnet? If this is what the machine.network.kubespan.filters.endpoints list is for, then I must be misunderstanding something, which seems likely, as I find the documentation for that setting pretty confusing.

[smira]

It's available in the machine config: https://www.talos.dev/v1.3/reference/configuration/#networkkubespan

You can filter advertised endpoints for each node to force them to use some network(s) you prefer.

[dhess]

Hi @smira, thanks for the reply.

Yes, sorry, as I mentioned, I am not able to get that setting to work as I expect. I might be misunderstanding it — perhaps I have it backwards?

Anyway, let me try an example. Let's say I have a ULA fdca:1e73:d078::/48, and each node in my cluster has an interface with an address assigned from subnet fdca:1e73:d078::64 in that ULA, and I want KubeSpan to use that subnet to connect to its peers. How would I specify that in filters.endpoints? I've tried something like this, but KubeSpan is still connecting on the public IPv4 subnet:

machine:
  network:
    kubespan:
      enabled: true
      # MTU must be physical MTU minus 80 bytes:
      # https://www.talos.dev/v1.3/kubernetes-guides/network/kubespan/#configuration
      mtu: 1320
      advertiseKubernetesNetworks: true
      allowDownPeerBypass: false
      filters:
        endpoints:
          - 10.0.0.0/8
          - fdca:1e73:d078::/48

(I've also added 10/8 as each of those network interfaces also has a 10.x.x.x/24 subnet IP, and that would also suffice.)

[smira]

you should leave only the networks you want KubeSpan to use.

you can verify that with talosctl get kubespanpeerspecs - it should list the list of advertised endpoints for each peer.

the running connection will keep using whatever endpoint it already picked, so you might need to trigger it to re-connect by e.g. rebooting a node.

[dhess]

Thanks, I managed to convince KubeSpan to use the 10.x.x.x/24 subnet using a config similar to the one above. I'm not sure why it didn't work the first few times, but it might have been a firewall issue.