IPAM to nmstate network config

[anmazzotti]

Part of #756

This PR should introduce network config support.
Note that the machine registration and reset flows still require DHCP support, however this PR removes the need of "infinite leases".

The elemental-operator will be able to use any CAPI IPAM provider to reserve and assign static IPAddresses to each MachineInventory.

The elemental-register will be able to ask for a network config to be applied. While the actual config in the NetworkTemplate is schemaless, this PR implements nmstate only.

This will require the OS to also have nmstate installed, so that the config can be applied.

Note that it is possible to define a NetworkTemplate on both MachineRegistration and MachineSelector. The latter will be applied during the MachineInventory adoption phase, so that it is possible to define IPPools per Cluster (rather per Registration only).

A registration example:

apiVersion: ipam.cluster.x-k8s.io/v1alpha2
kind: InClusterIPPool
metadata:
  name: elemental-inventory-pool
  namespace: fleet-default
spec:
  addresses:
    - 192.168.122.150-192.168.122.200
  prefix: 24
  gateway: 192.168.122.1
---
apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: fire-nodes
  namespace: fleet-default
spec:
  config:
    network:
      ipAddresses:
      - name: inventory-ip
        ipPoolRef:
          apiGroup: ipam.cluster.x-k8s.io
          kind: InClusterIPPool
          name: elemental-inventory-pool
      config:
        dns-resolver:
          config:
            server:
            - 192.168.122.1
            search: []
        routes:
          config:
          - destination: 0.0.0.0/0
            next-hop-interface: enp1s0
            next-hop-address: 192.168.122.1
            metric: 150
            table-id: 254
        interfaces:
          - name: enp1s0
            type: ethernet
            description: Main-NIC
            state: up
            ipv4:
              enabled: true
              dhcp: false
              address:
              - ip: "{inventory-ip}"
                prefix-length: 24
            ipv6:
              enabled: false

[kkaempf]

Can I express fixed IP assignments somehow ? 🤔

[anmazzotti]

The MachineInventory now carries complete information about the IPs and network config, so that on reconcile it's easier to manage everything.

apiVersion: v1
items:
  - apiVersion: elemental.cattle.io/v1beta1
    kind: MachineInventory
    metadata:
      annotations:
        elemental.cattle.io/network.applied: "false"
      name: test-btrfs-d47ca4cf-e95e-4e43-939b-e5bce460fbfc
      namespace: fleet-default
    spec:
      ipAddressClaims:
        inventory-ip:
          apiVersion: ipam.cluster.x-k8s.io/v1beta1
          kind: IPAddressClaim
          name: test-btrfs-d47ca4cf-e95e-4e43-939b-e5bce460fbfc-inventory-ip
          namespace: fleet-default
          uid: ac9d9629-c545-41ec-93f0-e7a59b2fcfd2
      ipAddressPools:
        - ipPoolRef:
            apiGroup: ipam.cluster.x-k8s.io
            kind: InClusterIPPool
            name: elemental-inventory-pool
          name: inventory-ip
      network:
        config:
          dns-resolver:
            config:
              search: []
              server:
                - 192.168.122.1
          interfaces:
            - description: Main-NIC
              ipv4:
                address:
                  - ip: '{inventory-ip}'
                    prefix-length: 24
                dhcp: false
                enabled: true
              ipv6:
                enabled: false
              name: enp1s0
              state: up
              type: ethernet
          routes:
            config:
              - destination: 0.0.0.0/0
                metric: 150
                next-hop-address: 192.168.122.1
                next-hop-interface: enp1s0
                table-id: 254
        ipAddresses:
          inventory-ip: 192.168.122.150
      tpmHash: 759d02b8cdfad743082ac528214a5b00cb3d6079aaa2b6bb35d87209e36f1fdb
    status:
      conditions:
        - lastTransitionTime: "2024-07-04T08:24:48Z"
          message: NetworkConfig is ready
          reason: ReconcilingNetworkConfig
          status: "True"
          type: NetworkConfigReady

[anmazzotti]

Can I express fixed IP assignments somehow ? 🤔

Not really. What are you trying to accomplish here?
There should be no particular attachment to one IP rather than another one, this is why we are supporting an IPAM provider.
You could theoretically define IPPools with 1 IP only, but that will also force you to one registration per machine and completely defeat what we are implementing here.

[kkaempf]

Can I express fixed IP assignments somehow ? 🤔

Not really. What are you trying to accomplish here?

I wonder how IP assignments to cluster nodes survive across reboots.

[anmazzotti]

I wonder how IP assignments to cluster nodes survive across reboots.

The network config is persisted through a yip file in /oem

For example this is included post-installation.
Note this will also run in recovery mode:

name: Apply network config
stages:
    initramfs:
        - files:
            - path: /oem/network/applied-config.yaml
              permissions: 384
              owner: 0
              group: 0
              content: |
                dns-resolver:
                  config:
                    search: []
                    server:
                    - 192.168.122.1
                interfaces:
                - description: Main-NIC
                  ipv4:
                    address:
                    - ip: '192.168.122.150'
                      prefix-length: 24
                    dhcp: false
                    enabled: true
                  ipv6:
                    enabled: false
                  name: enp1s0
                  state: up
                  type: ethernet
                routes:
                  config:
                  - destination: 0.0.0.0/0
                    metric: 150
                    next-hop-address: 192.168.122.1
                    next-hop-interface: enp1s0
                    table-id: 254
              encoding: ""
              ownerstring: ""
          directories:
            - path: /oem/network
              permissions: 448
              owner: 0
              group: 0
          if: '[ ! -f /oem/network/applied-config.yaml ]'
    network.before:
        - commands:
            - nmstatectl apply /oem/network/applied-config.yaml

[anmazzotti]

A lot happened in this PR.
The current status of things is that it does use nmstatectl to generate *.nmconnection files during the Elemental installation phase.

For example, given a MachineRegistration such as:

apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
metadata:
  name: fire-nodes
  namespace: fleet-default
spec:
  machineName: test-btrfs-${System Information/UUID}
  config:
    network:
      ipAddresses:
        inventory-ip:
          apiGroup: ipam.cluster.x-k8s.io
          kind: InClusterIPPool
          name: elemental-inventory-pool
      config:
        dns-resolver:
          config:
            server:
            - 192.168.122.1
            search: []
        routes:
          config:
          - destination: 0.0.0.0/0
            next-hop-interface: enp1s0
            next-hop-address: 192.168.122.1
            metric: 150
            table-id: 254
        interfaces:
          - name: enp1s0
            type: ethernet
            description: Main-NIC
            state: up
            ipv4:
              enabled: true
              dhcp: false
              address:
              - ip: "{inventory-ip}"
                prefix-length: 24
            ipv6:
              enabled: false

Will lead to the following yip file (/oem/94_custom.yaml) to apply network configuration:

name: Apply network config
stages:
    initramfs:
        - files:
            - path: /etc/NetworkManager/system-connections/Wired connection 1.nmconnection
              permissions: 384
              owner: 0
              group: 0
              content: |
                [connection]
                id=Wired connection 1
                uuid=d26b4ae4-d525-3cbf-a557-33feb60343c0
                type=ethernet
                autoconnect-priority=-999
                interface-name=enp1s0
                timestamp=1722339542

                [ethernet]

                [ipv4]
                address1=192.168.122.150/24
                dhcp-timeout=2147483647
                dns=192.168.122.1;
                dns-options=
                dns-priority=40
                method=manual
                route1=0.0.0.0/0,192.168.122.1,150
                route1_options=table=254

                [ipv6]
                addr-gen-mode=eui64
                dhcp-timeout=2147483647
                method=disabled

                [proxy]

                [user]
                nmstate.interface.description=Main-NIC
              encoding: ""
              ownerstring: ""
          if: '[ -f /run/elemental/active_mode ]'

[fgiudici]

Don't we already have networkConfigData as part of data (and yaml unmarshalled cfg) above?
data should be the full MachineRegistration elemental config, including the networking part (if not there maybe we have to explicitly add on the operator side).
We may save extending the websocket protocol here

[anmazzotti]

The data contains the MachineRegistration data, but this will not contain the real IPAddresses.

I want to avoid replacing this data (and wait for real IPs) at this stage, this is why I think it's better to have a more explicit request.
Extending the protocol is not a problem I believe, but definitely we can improve in the future by not closing the websocket connection.

[fgiudici]

We will likely still rework this in few ways, also changing the API, but since this is working and already a pretty good starting point, let's merge it 🎉