Can't get image cache working from disk image with qemu cluster

[alongwill]

I used imager v1.9.2 on x86_64 to create a disk image: metal-amd64.raw.zst (470MB). The images.txt file used was:

ghcr.io/siderolabs/flannel:v0.26.1
registry.k8s.io/coredns/coredns:v1.12.0
gcr.io/etcd-development/etcd:v3.5.17
registry.k8s.io/kube-apiserver:v1.32.0
registry.k8s.io/kube-controller-manager:v1.32.0
registry.k8s.io/kube-scheduler:v1.32.0
registry.k8s.io/kube-proxy:v1.32.0
ghcr.io/siderolabs/kubelet:v1.32.0
ghcr.io/siderolabs/installer:v1.9.1
registry.k8s.io/pause:3.10

I created the cluster using:

sudo -E talosctl cluster create \
    --name=laptop-qemu \
    --provisioner=qemu --cidr=172.20.0.0/24 --install-image=ghcr.io/siderolabs/installer:v1.9.1 --memory 2048 --memory-workers 2048 --disk 6144 --cpus 2 --controlplanes 1 --workers 1 \
    --skip-injecting-config --wait=false --with-init-node \
    --disk-image-path=_out/metal-amd64.raw.zst

In the controlplane.yaml I'm adding:

machine:
  features:
    imageCache:
      localEnabled: true

and

apiVersion: v1alpha1
kind: VolumeConfig
name: IMAGECACHE
provisioning:
  diskSelector:
    match: 'system_disk'
  minSize: 2GB
  maxSize: 2GB

After running the apply-config this shows in ~/.talos/clusters/laptop-qemu/laptop-qemu-controlplane-1.log

[  110.534825] [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "STATE", "phase": "missing -> failed", "error": "error evaluating disk locator: no such attribute(s): system_disk"}
[  110.543443] [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "IMAGECACHE-ISO", "phase": "waiting -> missing"}
[  110.552075] [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "STATE", "phase": "failed -> failed", "error": "error evaluating disk locator: no such attribute(s): system_disk"}
[  110.564110] [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "STATE", "phase": "failed -> failed", "error": "error evaluating disk locator: no such attribute(s): system_disk"}

What have I done wrong? 😅

[smira]

--disk-image-path should point to uncompressed (raw) image

[alongwill]

When I use the uncompressed image .talos/clusters/laptop-qemu/laptop-qemu-controlplane-1.log shows the GRUB output.

After applying the config in the dashboard I see logs like:

user: warning: [2025-02-10T13:03:58.669091527Z]: [talos] controller failed {"component": "controller-runtime", "controller": "k8s.NodeApplyController", "error": "1 error(s) occurred:
 \n\ttimeout"}
 user: warning: [2025-02-10T13:04:15.621129527Z]: [talos] kubernetes endpoint watch error {"component": "controller-runtime", "controller": "k8s.EndpointController", "error": "failed to
 list *v1.Endpoints: Get \"https://172.20.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&limit=500&resourceVersion=0\": EOF"}
 user: warning: [2025-02-10T13:04:19.425955527Z]: [talos] volume status {"component": "controller-runtime", "controller": "block.VolumeManagerController", "volume": "STATE", "phase":
 "failed -> failed", "error": "no disks matched for volume"}

Full dmesg output here: https://gist.github.com/alongwill/e97e8f1a8ff14f9ca20560896cae1285

talosctl get imagecacheconfig shows:

node: 172.20.0.2
metadata:
    namespace: cri
    type: ImageCacheConfigs.cri.talos.dev
    id: image-cache
    version: 4
    owner: cri.ImageCacheConfigController
    phase: running
    created: 2025-02-10T11:27:20Z
    updated: 2025-02-10T11:30:44Z
spec:
    status: ready
    copyStatus: skipped
    roots:
        - /system/imagecache/disk

talosctl get volumestatuses shows:

NODE         NAMESPACE   TYPE           ID               VERSION   PHASE     LOCATION    SIZE
172.20.0.2   runtime     VolumeStatus   EPHEMERAL        1         waiting
172.20.0.2   runtime     VolumeStatus   IMAGECACHE       1         ready     /dev/vda5   9.6 GB
172.20.0.2   runtime     VolumeStatus   IMAGECACHE-ISO   1         missing
172.20.0.2   runtime     VolumeStatus   META             2         ready     /dev/vda4   1.0 MB
172.20.0.2   runtime     VolumeStatus   STATE            3         failed

Image cache files appear to be present on disk:

talosctl ls /system/imagecache/disk/manifests/ghcr.io/siderolabs
NODE         NAME
172.20.0.2   .
172.20.0.2   flannel
172.20.0.2   installer
172.20.0.2   kubelet

talosctl services shows:

NODE         SERVICE      STATE     HEALTH   LAST CHANGE    LAST EVENT
172.20.0.2   apid         Running   OK       1h49m2s ago    Health check successful
172.20.0.2   auditd       Running   OK       1h52m25s ago   Health check successful
172.20.0.2   containerd   Running   OK       1h52m25s ago   Health check successful
172.20.0.2   dashboard    Running   ?        1h52m21s ago   Process Process(["/sbin/dashboard"]) started with PID 1925
172.20.0.2   machined     Running   OK       1h52m25s ago   Health check successful
172.20.0.2   registryd    Running   OK       1h49m1s ago    Health check successful
172.20.0.2   syslogd      Running   OK       1h52m24s ago   Health check successful
172.20.0.2   udevd        Running   OK       1h52m24s ago   Health check successfu

talosctl get members doesn't return anything:

NODE   NAMESPACE   TYPE   ID   VERSION   HOSTNAME   MACHINE TYPE   OS   ADDRESSES

I tried bootstrap but it failed

error executing bootstrap: 1 error occurred:
        * 172.20.0.2: rpc error: code = FailedPrecondition desc = bootstrap is not available yet

[smira]

So look at your /dev/vda size now, I guess it doesn't have enough space to allocate STATE/EPHEMERAL, as I guess you have them by default pinned to the system disk.

[alongwill]

I increased --disk from 6144 to 16144. All working now 🎉 Thanks!