retrying error: invalid configuration: unable to read client-cert /var/lib/kubelet/pki/kubelet-client-current.pem ... no such file or directory

[bcspragu]

Hi there,

I'm setting up a Talos cluster on bare metal on a few blade servers. I've got the first control node up, but the first worker node gets stuck with the error in the title, and continuously reboots. I haven't changed the talosctl-generated configs much except to add static IPs, enable disk encryption, and use K8s discovery.

Snippet of `talos logs kubelet` on the worker

192.168.3.4: {"ts":1676598555766.3599,"caller":"kubelet/kubelet_node_status.go:70","msg":"Attempting to register node","v":0,"node":{"name":"talos-o4k-hx7"}}
192.168.3.4: {"ts":1676598555767.4775,"caller":"kubelet/kubelet_node_status.go:92","msg":"Unable to register node with API server","node":{"name":"talos-o4k-hx7"},"err":"Unauthorized"}
192.168.3.4: {"ts":1676598557042.345,"caller":"cache/reflector.go:424","msg":"vendor/k8s.io/client-go/informers/factory.go:150: failed to list *v1.Node: Unauthorized\n","v":0}
192.168.3.4: {"ts":1676598557042.396,"caller":"cache/reflector.go:140","msg":"vendor/k8s.io/client-go/informers/factory.go:150: Failed to watch *v1.Node: failed to list *v1.Node: Unauthorized\n"}
192.168.3.4: {"ts":1676598562088.5315,"caller":"eviction/eviction_manager.go:261","msg":"Eviction manager: failed to get summary stats","err":"failed to get node info: node \"talos-o4k-hx7\" not found"}
192.168.3.4: {"ts":1676598562627.988,"caller":"lease/controller.go:146","msg":"failed to ensure lease exists, will retry in 7s, error: Unauthorized\n"}

controlplane.yaml (the useful bits)

version: v1alpha1
debug: false
persist: true
machine:
    type: controlplane
    token: ...
    ca:
        crt: ...
        key: ...
    certSANs: []
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.26.1
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    network:
      interfaces:
        - interface: eth0
          addresses:
            - 192.168.3.3/24
          routes:
            - network: 0.0.0.0/0
              gateway: 192.168.3.1
          vip:
            ip: 192.168.3.19
      nameservers:
          - 8.8.8.8
    install:
        disk: /dev/sda
        image: ghcr.io/siderolabs/installer:v1.3.3
        bootloader: true
        wipe: false
    registries: {}
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
    systemDiskEncryption:
        ephemeral:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0
        state:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0
cluster:
    id: ...
    secret: ...
    controlPlane:
        endpoint: https://192.168.3.19:6443
    clusterName: m1000e
    network:
        dnsDomain: cluster.local
        podSubnets:
            - 10.244.0.0/16
        serviceSubnets:
            - 10.96.0.0/12
    discovery:
        enabled: true
        registries:
            kubernetes: {}
    [... omitted a bunch of auto-generated and untouched config ...]

worker.yaml

version: v1alpha1
debug: false
persist: true
machine:
    type: worker
    token: ...
    ca:
        crt: ...
        key: ""
    certSANs: []
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.26.1
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    network:
      interfaces:
        - interface: eth0
          addresses:
            - 192.168.3.4/24
          routes:
            - network: 0.0.0.0/0
              gateway: 192.168.3.1
      nameservers:
          - 8.8.8.8
    install:
        disk: /dev/sda
        image: ghcr.io/siderolabs/installer:v1.3.3
        bootloader: true
        wipe: false
    registries: {}
    features:
        rbac: true
        stableHostname: true
        apidCheckExtKeyUsage: true
    systemDiskEncryption:
        ephemeral:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0
        state:
            provider: luks2
            keys:
                - nodeID: {}
                  slot: 0
cluster:
    id: ...
    secret: ...
    controlPlane:
        endpoint: https://192.168.3.19:6443
    network:
        dnsDomain: cluster.local
        podSubnets:
            - 10.244.0.0/16
        serviceSubnets:
            - 10.96.0.0/12
    ca:
        crt: ...
        key: ""
    discovery:
        enabled: true
        registries:
            kubernetes: {}

[smira]

The error in the end means that kubelet on the worker node can't join the controlplane. The exact reason is hard to guess, but it might be bootstrap token being wrong? controlplane endpoint being wrong? ...

Kubelet uses bootstrap token to provision the initial client certificate and register the node.

[bcspragu]

Hey smira, thanks for the answer!

The control plane endpoint is correct, because I remember seeing logs (which I unfortunately did not capture) on the control plane saying that it had denied the request as well, so the controlplane <-> worker link is up.

The bootstrap token being wrong is an interesting idea. Theoretically, these nodes were created against configs from the same secrets.yaml file, but I had gone through the process (including generating secrets) a few times, so perhaps things got out of sync at some point. I'll try again from scratch and see what happens, thanks!