Kubelet fails on Docker CE in GCP Compute Engine VM #5073

scenox · 2022-03-03T12:49:57Z

scenox
Mar 3, 2022

I've installed docker-ce on a GCP Compute Engine Ubuntu 18.04LTE VM instance and tried to bootstrap the cluster:

$ talosctl cluster create --wait
validating CIDR and reserving IPs
generating PKI and tokens
downloading ghcr.io/talos-systems/talos:v0.14.2
creating network talos-default
creating master nodes
creating worker nodes
waiting for API
bootstrapping cluster
waiting for etcd to be healthy: OK
waiting for apid to be ready: OK
◱ waiting for kubelet to be healthy: 1 error occurred:
    * 10.5.0.2: service is not healthy: kubelet
 context deadline exceeded

Cluster:

$ talosctl cluster show
PROVISIONER       docker
NAME              talos-default
NETWORK NAME      talos-default
NETWORK CIDR      10.5.0.0/24
NETWORK GATEWAY   <nil>
NETWORK MTU       1500

NODES:

NAME                     TYPE           IP         CPU   RAM   DISK
talos-default-master-1   controlplane   10.5.0.2   -     -     -
talos-default-worker-1   worker         10.5.0.3   -     -     -

$ talosctl -n talos-default-master-1 services
NODE                     SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT
talos-default-master-1   apid         Running   OK       6m53s ago     Health check successful
talos-default-master-1   containerd   Running   OK       6m54s ago     Health check successful
talos-default-master-1   cri          Running   OK       6m53s ago     Health check successful
talos-default-master-1   etcd         Running   OK       6m26s ago     Health check successful
talos-default-master-1   kubelet      Running   Fail     6m24s ago     Health check failed: Get "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused
talos-default-master-1   machined     Running   ?        6m55s ago     Service started as goroutine
talos-default-master-1   trustd       Running   OK       6m53s ago     Health check successful

$ docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57
 Built:             Mon Dec 13 11:45:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.12
  Git commit:       459d0df
  Built:            Mon Dec 13 11:43:36 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Last lines kubelet log:

talos-default-master-1: {"ts":1646311657292.283,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
talos-default-master-1: {"ts":1646311657292.3442,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
talos-default-master-1: {"ts":1646311666470.6892,"caller":"certificate/transport.go:112","msg":"No valid client certificate is found but the server is not responsive. A restart may be necessary to retrieve new initial credentials.","lastCertificateAvailabilityTime":1646311666470.6992,"shutdownThreshold":"5m0s"}

Last lines machined log:

talos-default-master-1: [talos] 2022/03/03 12:44:48 controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Secret/bootstrap-token-965enb: Get \"https://localhost:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"}
talos-default-master-1: [talos] 2022/03/03 12:44:57 controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \"https://127.0.0.1:10250/pods/?timeout=30s\": dial tcp 127.0.0.1:10250: connect: connection refused"}
talos-default-master-1: machined/authz/injector 2022/03/03 12:45:33 parsed metadata [os:admin] as [os:admin] (unknownRoles = [])
talos-default-master-1: machined/authz/authorizer 2022/03/03 12:45:33 authorized ([os:admin os:reader] includes [os:admin])
talos-default-master-1: machined 2022/03/03 12:45:33 OK [/machine.MachineService/Logs] 382.97µs stream Success (:authority=localhost;content-type=application/grpc;proxyfrom=10.5.0.2:50000;talos-role=os:admin;user-agent=grpc-go/1.42.0)
talos-default-master-1: machined/authz/injector 2022/03/03 12:45:48 parsed metadata [os:admin] as [os:admin] (unknownRoles = [])
talos-default-master-1: machined/authz/authorizer 2022/03/03 12:45:48 authorized ([os:admin os:reader] includes [os:admin])
talos-default-master-1: machined 2022/03/03 12:45:48 OK [/machine.MachineService/Logs] 248.646µs stream Success (:authority=localhost;content-type=application/grpc;proxyfrom=10.5.0.2:50000;talos-role=os:admin;user-agent=grpc-go/1.42.0)
talos-default-master-1: [talos] 2022/03/03 12:46:10 controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Secret/bootstrap-token-965enb: Get \"https://localhost:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"}
talos-default-master-1: [talos] 2022/03/03 12:46:12 controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \"https://127.0.0.1:10250/pods/?timeout=30s\": dial tcp 127.0.0.1:10250: connect: connection refused"}
talos-default-master-1: [talos] 2022/03/03 12:47:06 controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \"https://127.0.0.1:10250/pods/?timeout=30s\": dial tcp 127.0.0.1:10250: connect: connection refused"}
talos-default-master-1: [talos] 2022/03/03 12:47:33 controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Secret/bootstrap-token-965enb: Get \"https://localhost:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"}
talos-default-master-1: machined/authz/injector 2022/03/03 12:47:52 parsed metadata [os:admin] as [os:admin] (unknownRoles = [])
talos-default-master-1: machined/authz/authorizer 2022/03/03 12:47:52 authorized ([os:admin os:reader] includes [os:admin])
talos-default-master-1: machined 2022/03/03 12:47:52 OK [/machine.MachineService/Logs] 487.393µs stream Success (:authority=localhost;content-type=application/grpc;proxyfrom=10.5.0.2:50000;talos-role=os:admin;user-agent=grpc-go/1.42.0)
talos-default-master-1: [talos] 2022/03/03 12:48:09 controller failed {"component": "controller-runtime", "controller": "k8s.KubeletStaticPodController", "error": "error refreshing pod status: error fetching pod status: Get \"https://127.0.0.1:10250/pods/?timeout=30s\": dial tcp 127.0.0.1:10250: connect: connection refused"}
talos-default-master-1: [talos] 2022/03/03 12:48:15 controller failed {"component": "controller-runtime", "controller": "k8s.ManifestApplyController", "error": "error creating mapping for object /v1/Secret/bootstrap-token-965enb: Get \"https://localhost:6443/api?timeout=32s\": dial tcp 127.0.0.1:6443: connect: connection refused"}
talos-default-master-1: machined/authz/injector 2022/03/03 12:48:54 parsed metadata [os:admin] as [os:admin] (unknownRoles = [])
talos-default-master-1: machined/authz/authorizer 2022/03/03 12:48:54 authorized ([os:admin os:reader] includes [os:admin])

smira · 2022-03-03T13:25:11Z

smira
Mar 3, 2022
Maintainer

can you please attach full docker logs talos-default-master-1 please?

0 replies

scenox · 2022-03-03T13:50:42Z

scenox
Mar 3, 2022
Author

Same issue occurs also with Debian and Docker CE on Google Compute Engine. Logs attached:
docker-logs-master.txt
Ubuntu logs:
docker-logs-master-ubuntu.txt
This looks suspicious:

[talos] 2022/03/03 14:13:08 service[etcd](Failed): Failed to run pre stage: failed to pull image "gcr.io/etcd-development/etcd:v3.5.1": 1 error(s) occurred:
        failed to pull image "gcr.io/etcd-development/etcd:v3.5.1": failed to copy: httpReadSeeker: failed open: failed to do request: context canceled

1 reply

smira Mar 3, 2022
Maintainer

So looking at the logs, it's certainly something with the kubelet. The error above is fine, image pull is aborted by bootstrap, but it finishes fine after that.

Most probably we'll need full talosctl logs kubelet. And by the way, what is the talosctl version?

scenox · 2022-03-03T15:11:29Z

scenox
Mar 3, 2022
Author

$ talosctl -n 10.5.0.2 version
Client:
        Tag:         v0.14.2
        SHA:         06ea409b
        Built:       
        Go version:  go1.17.7
        OS/Arch:     linux/amd64
Server:
        NODE:        10.5.0.2
        Tag:         v0.14.2
        SHA:         06ea409b
        Built:       
        Go version:  go1.17.7
        OS/Arch:     linux/amd64
        Enabled:     RBAC

$ talosctl -n 10.5.0.2 logs kubelet
10.5.0.2: {"ts":1646320090382.6887,"caller":"app/server.go:446","msg":"Kubelet version","v":0,"kubeletVersion":"v1.23.1"}
10.5.0.2: {"ts":1646320090383.4832,"caller":"app/server.go:874","msg":"Client rotation is on, will bootstrap in background","v":0}
10.5.0.2: {"ts":1646320090388.8542,"caller":"dynamiccertificates/dynamic_cafile_content.go:156","msg":"Starting controller","v":0,"name":"client-ca-bundle::/etc/kubernetes/pki/ca.crt"}
10.5.0.2: {"ts":1646320090390.8884,"caller":"certificate/certificate_manager.go:471","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://10.5.0.2:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320092528.9543,"caller":"certificate/certificate_manager.go:471","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://10.5.0.2:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320095404.0098,"caller":"cm/container_manager_linux.go:281","msg":"Container manager verified user specified cgroup-root exists","v":0,"cgroupRoot":[]}
10.5.0.2: {"ts":1646320095404.0767,"caller":"cm/container_manager_linux.go:286","msg":"Creating Container Manager object based on Node Config","v":0,"nodeConfig":{"RuntimeCgroupsName":"","SystemCgroupsName":"/system","KubeletCgroupsName":"/podruntime/kubelet","ContainerRuntime":"remote","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"cgroupfs","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":false,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"50m","ephemeral-storage":"256Mi","memory":"128Mi","pid":"100"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"ExperimentalCPUManagerPolicy":"none","ExperimentalCPUManagerPolicyOptions":null,"ExperimentalTopologyManagerScope":"container","ExperimentalCPUManagerReconcilePeriod":10000000000,"ExperimentalMemoryManagerPolicy":"None","ExperimentalMemoryManagerReservedMemory":null,"ExperimentalPodPidsLimit":-1,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"ExperimentalTopologyManagerPolicy":"none"}}
10.5.0.2: {"ts":1646320095404.4656,"caller":"topologymanager/topology_manager.go:133","msg":"Creating topology manager with policy per scope","v":0,"topologyPolicyName":"none","topologyScopeName":"container"}
10.5.0.2: {"ts":1646320095404.4873,"caller":"cm/container_manager_linux.go:321","msg":"Creating device plugin manager","v":0,"devicePluginEnabled":true}
10.5.0.2: {"ts":1646320095404.6548,"caller":"state/state_mem.go:36","msg":"Initialized new in-memory state store","v":0}
10.5.0.2: {"ts":1646320095416.1252,"caller":"kubelet/kubelet.go:416","msg":"Attempting to sync node with API server","v":0}
10.5.0.2: {"ts":1646320095416.3276,"caller":"kubelet/kubelet.go:278","msg":"Adding static pod path","v":0,"path":"/etc/kubernetes/manifests"}
10.5.0.2: {"ts":1646320095416.4897,"caller":"kubelet/kubelet.go:289","msg":"Adding apiserver pod source","v":0}
10.5.0.2: {"ts":1646320095416.6462,"caller":"config/apiserver.go:42","msg":"Waiting for node sync before watching apiserver pods","v":0}
10.5.0.2: {"ts":1646320095418.9236,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320095419.0134,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320095419.3718,"caller":"kuberuntime/kuberuntime_manager.go:248","msg":"Container runtime initialized","v":0,"containerRuntime":"containerd","version":"v1.5.8","apiVersion":"v1alpha2"}
10.5.0.2: {"ts":1646320095420.412,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320095420.4548,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320096239.0874,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320096239.3442,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320096912.408,"caller":"certificate/certificate_manager.go:471","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://10.5.0.2:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320096920.2505,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320096920.3252,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320099429.1196,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320099429.1865,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320100042.1155,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320100042.2854,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320105062.5613,"caller":"certificate/certificate_manager.go:471","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://10.5.0.2:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320105246.5452,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320105246.6775,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320105364.5132,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320105364.616,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320112476.255,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320112476.3032,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320117846.476,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320117846.5527,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320121394.7122,"caller":"certificate/certificate_manager.go:471","msg":"kubernetes.io/kube-apiserver-client-kubelet: Failed while requesting a signed certificate from the control plane: cannot create certificate signing request: Post \"https://10.5.0.2:6443/apis/certificates.k8s.io/v1/certificatesigningrequests\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320121394.762,"caller":"certificate/certificate_manager.go:354","msg":"kubernetes.io/kube-apiserver-client-kubelet: Reached backoff limit, still unable to rotate certs: timed out waiting for the condition\n"}
10.5.0.2: {"ts":1646320131914.747,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320131914.8054,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get \"https://10.5.0.2:6443/api/v1/services?limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}
10.5.0.2: {"ts":1646320137864.2368,"caller":"cache/reflector.go:324","msg":"k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n","v":0}
10.5.0.2: {"ts":1646320137864.2935,"caller":"cache/reflector.go:138","msg":"k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get \"https://10.5.0.2:6443/api/v1/nodes?fieldSelector=metadata.name%3Dtalos-default-master-1&limit=500&resourceVersion=0\": dial tcp 10.5.0.2:6443: connect: connection refused\n"}

1 reply

smira Mar 3, 2022
Maintainer

I don't see anything fatal here. It looks like kubelet never boots successfully, doesn't become healthy, never runs static pods for the control plane, so the cluster never bootstraps. But I don't see any root cause here.

scenox · 2022-03-03T16:08:26Z

scenox
Mar 3, 2022
Author

Not sure what is going on on Google Compute Engine but it works without any issues on a MS Azure instance with exactly same OS (Ubuntu 18.04.06) and same docker ce version.

1 reply

maxpain Aug 15, 2022

The same problem on GCP.

gpi-atos · 2023-03-18T17:44:32Z

gpi-atos
Mar 18, 2023

I have the same issue trying to make a local talos environment in Docker CE running in GCE. I did try a few things:

the list of images I tried: Ubuntu 20.04 & 22.04, Debian 11, CentOS Stream 8
I tried to tweak the OS with those kernel modules:

sudo sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
---
lsmod | grep br_netfilter
lsmod | grep overlay
br_netfilter           32768  0
bridge                253952  1 br_netfilter
overlay               147456  5

allowed iptables to accept all traffic.
deleted /etc/containerd/config.toml
added /etc/docker/daemon.js
I tried the talos v0.9.1, v1.2.6 and latest as of now (v1.3.4)
So far I couldn't make it past this error : [talos] 2023/03/18 15:25:14 service[kubelet](Running): Health check failed: Get "http://127.0.0.1:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused

The only issue I managed to tackle was one when I was playing wih the v0.9.1: talos couldn't fetch kubelet because of a TLS Handshake error. The proper way to handle that is to add talosctl cluster create --mtu 1460 (if your vpc underneath your GCE is configured that way).
So what's next? Maybe testing with different Docker CE version.

1 reply

gpi-atos Mar 19, 2023

Tested on Fedora CoreOS 37 stable: same issue.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kubelet fails on Docker CE in GCP Compute Engine VM #5073

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Kubelet fails on Docker CE in GCP Compute Engine VM #5073

scenox Mar 3, 2022

Replies: 5 comments · 4 replies

smira Mar 3, 2022 Maintainer

scenox Mar 3, 2022 Author

smira Mar 3, 2022 Maintainer

scenox Mar 3, 2022 Author

smira Mar 3, 2022 Maintainer

scenox Mar 3, 2022 Author

maxpain Aug 15, 2022

gpi-atos Mar 18, 2023

gpi-atos Mar 19, 2023

scenox
Mar 3, 2022

Replies: 5 comments 4 replies

smira
Mar 3, 2022
Maintainer

scenox
Mar 3, 2022
Author

smira Mar 3, 2022
Maintainer

scenox
Mar 3, 2022
Author

smira Mar 3, 2022
Maintainer

scenox
Mar 3, 2022
Author

gpi-atos
Mar 18, 2023