How do i get debug logs out of etcd?

[Hobbit44]

I'm trying to migrate from k0s to talos and at the moment i cant get a single talos control plane to join. Im trying to find out what etcd is complaining about but i cant find anything resembling errors or anything. I can only see repeated service "etcd" to be "up".

Iv tried enabling the debug: true on the machine config, Adding log-level=debug through the extraArgs and various other things to the k0s to expose/verify/authorize, but to no avail. My biggest issue is that i can see whats happening on the talos side. Im just fumbling around in the dark hoping to stumble upon the solution.

Any advice would be greatly appreciated.

[smira]

You can do talosctl logs etcd to see the logs of etcd (or any other service), but as it's not running yet, it won't work.

Please don't do debug: true, this might lead to other fun issue.

You should look at the console logs, explain what you're trying to do specifically, as it's not clear.

[Hobbit44]

Specifically, I have an existing cluster running on proxmox that was deployed using k0s, consisting of 3 control planes operating as workers too, and 3 additional workers. At the moment I have an additional VM with talos installed that im trying to add as an additional control plane with the intention of making that the only one. I've followed the guide for migrating from kubeadm as its the closest i could find to a guide for integrating talos into an existing cluster.

As far as logs go, after the initial bootup sequence, the reconcile loop swings through error refreshing pod status connection refused relating to kube-api not being started, which as far as i see is expected as etcd hasnt started. Etcd only seems to produce the message i mentioned above as well as saying waiting to join.

I've checked for overlapping pod and service cidrs. I've adjusted the k0s settings to make it more open, matching as many args from k0s into the extraArgs on talos. K0s uses jwt auth tokens which i have tried added to talos but that didnt seem to work, but also i cant tell because there doesnt seem to be a way to get access to logs for etcd or control the log level of components. Anything im missing?

[smira]

I don't have any information on migrating from k0s.

Talos will try to learn etcd IPs by contacting Kubernetes controlplane endpoint (as you configured it), and then join the cluster using etcd PKI (you submitted as well).

Errors on join should be reported via retrying: errors in the console logs.

[Hobbit44]

There doesn't seem to any reference to retrying in the console output apart from retrying error: error building etcd client TLS config: open /system/secrets/etcd/admin.crt: no such file or directory which disappears after a little so i'm attributing it to part of the boot process as the file is present. Which component does the communicating? I don't see anything in the logs about a communication to my configured endpoint. I only see the messages about being unable to connect to localhost:6443 which seems to be the address in the bootstrap-kubeconfig which i understand is expected.

[smira]

If you need help, please submit machine configuration and full kernel (console) logs.

[Hobbit44]

Ok. i managed to get it solved but it was a faff. @smira So your aware. (for context im trying to move from HA to single CP)

In the end i had to to overwrite the k0s config by adding the following to /etc/k0s/k0s.yaml. Matching the ips for the control plane nodes. In my case i have 3, with ips ending in 31,32, and 33:

spec:
  storage:
    etcd:
      extraArgs:
        initial-advertise-peer-urls: https://10.0.1.31:2380
        advertise-client-urls: http://10.0.1.31:2379
        listen-client-urls: https://0.0.0.0:2379
        listen-peer-urls: https://0.0.0.0:2380

Then run systemctl restart k0scontroller to restart the node and etcd. You can then watch the logs with journalctl -u k0scontroller -f -n 100 -g "component=etcd".

I then ran this to add the new node to the member list (etcdctl is available in the releases package from etcd):

etcdctl --key=/var/lib/k0s/pki/etcd/ca.key --cert=/var/lib/k0s/pki/etcd/ca.crt --cacert=/var/lib/k0s/pki/etcd/ca.crt member add kube-control --peer-urls=https://10.0.1.20:2380

On the talos side i have the following for etcd (in the docs it say initial-cluster-state is not allowed but it accepted it. I have no idea if it helped or not 🤷):

cluster:
  etcd:
    extraArgs:
      initial-cluster-state: existing
      initial-cluster: "kube-control=https://10.0.1.20:2380,kube-c1=https://10.0.1.31:2380,kube-c2=https://10.0.1.32:2380,kube-c3=https://10.0.1.33:2380"
      initial-advertise-peer-urls: https://10.0.1.20:2380

This seemed to get everything connected and now the node is online, part of the cluster and it looks like im good to go. Some things to note are:

• I had auth-token: jwt,pub-key=/var/lib/pki/etcd/jwt.pub,priv-key=/var/lib/pki/etcd/jwt.key,sign-method=RS512,ttl=10m in the talos etcd extra args when it first connected but it imediately started complaining that it was missing the cert files. not unnexpected as i had know idea how they are created (im a little inexperienced with etcd), but after removing it and clearing the ephemeral partition with talosctl -n <IP> reset --graceful=false --reboot --system-labels-to-wipe=EPHEMERAL it connected. Im assuming its not needed as i believe that clears all the etcd config on disk but im just noting things down anyway.
• On th k0s side i ran without the advertise-client-urls initially and my kube-c2 node decided to regenerate its peer certs with a SAN of 10.2.4.83 which made it throw remote error: tls: bad certificate and all the other nodes throw tls: failed to verify certificate: x509: certificate is valid for 10.2.4.1, not 10.0.1.32. For the life of me i dont know why it did that but its getting binned anyway so i dont think i care. I just stopped k0s on it and I'l deal with it tomorrow as its 1am.

This is my full talos config:

control-plane.yaml

version: v1alpha1
persist: true
machine:
  type: controlplane
  token: [redacted]
  ca:
    crt: [redacted]
    key: [redacted]
  certSANs: 
    - [redacted]
  kubelet:
    image: ghcr.io/siderolabs/kubelet:v1.32.0
    defaultRuntimeSeccompProfileEnabled: true
    disableManifestsDirectory: true
  network: 
    hostname: kube-control
  install:
    disk: /dev/sda
    image: ghcr.io/siderolabs/installer:v1.9.1
    wipe: false
    extraKernelArgs:
      - gfxpayload=1280x1024x24
      - gfxmode=1280x1024
      - console=tty0
      - console=ttyS0
  registries: {}
  features:
    rbac: true
    stableHostname: true
    apidCheckExtKeyUsage: true
    diskQuotaSupport: true
    hostDNS:
      enabled: true
      forwardKubeDNSToHost: true
    kubernetesTalosAPIAccess:
      enabled: true
      allowedRoles:
        - os:reader
      allowedKubernetesNamespaces:
        - kube-system
        - system
  nodeLabels:
    node.kubernetes.io/exclude-from-external-load-balancers: ""
cluster:
  id: [redacted]
  secret: [redacted]
  controlPlane:
    endpoint: https://[redacted]:6443
  clusterName: home-k8s-cluster
  network:
    dnsDomain: cluster.local
    podSubnets:
      - 10.2.0.0/16
    serviceSubnets:
      - 10.20.0.0/16
    cni:
      name: none
  token: [redacted]
  ca:
    crt: [redacted]
    key: [redacted]
  aggregatorCA:
    crt: [redacted]
    key: [redacted]
  serviceAccount:
    key: [redacted]
  apiServer:
    image: registry.k8s.io/kube-apiserver:v1.32.0
    certSANs:
      - [redacted]
    disablePodSecurityPolicy: true
  controllerManager:
    image: registry.k8s.io/kube-controller-manager:v1.32.0
  proxy:
    image: registry.k8s.io/kube-proxy:v1.32.0
  scheduler:
    image: registry.k8s.io/kube-scheduler:v1.32.0
  discovery:
    enabled: true
    registries:
      kubernetes:
        disabled: false
      service: 
        disabled: true 
  etcd:
    image: gcr.io/etcd-development/etcd:v3.5.12
    extraArgs:
      initial-cluster-state: existing
      initial-cluster: "kube-control=https://10.0.1.20:2380,kube-c1=https://10.0.1.31:2380,kube-c2=https://10.0.1.32:2380,kube-c3=https://10.0.1.33:2380"
      initial-advertise-peer-urls: https://10.0.1.20:2380
    ca:
      crt: [redacted]
      key: [redacted]
  extraManifests: []
  inlineManifests: []

And my k0s config from kube-c1 in case its useful:

k0s.yaml

Metadata:
  name: null
apiVersion: ""
kind: ""
spec:
  api:
    address: 10.0.1.31
    externalAddress: [redacted]
    sans:
    - 10.0.1.31
    - [redacted]
    - 10.0.1.32
    - 10.0.1.33
    - 127.0.0.1
  network:
    kubeProxy:
      iptables:
        masqueradeAll: false
        masqueradeBit: null
        minSyncPeriod: 0s
        syncPeriod: 0s
    kuberouter:
      autoMTU: true
      mtu: 0
      peerRouterASNs: ""
      peerRouterIPs: ""
    podCIDR: 10.2.0.0/16
    serviceCIDR: 10.20.0.0/16
  storage:
    etcd:
      extraArgs:
        initial-advertise-peer-urls: https://10.0.1.31:2380
        advertise-client-urls: http://10.0.1.31:2379
        listen-client-urls: https://0.0.0.0:2379
        listen-peer-urls: https://0.0.0.0:2380