Unable to Fetch ECR Images from On-Prem Machine Despite Installing ecr-credential-provider Extension

[vitamaxDH]

Hey everyone, I'm struggling to set up the ecr-credential-provider.

Here’s what I’ve tried so far:

1. Installed Talos.
2. Upgraded with ecr-credential-provider:

t upgrade --image factory.talos.dev/installer/10e276a06c1f86b182757a962258ac00655d3425e5957f617bdc82f06894e39b:v1.8.2 -m powercycle -f

3. Applied the following patch:

---
machine:
  kubelet:
    credentialProviderConfig:
      apiVersion: kubelet.config.k8s.io/v1
      kind: CredentialProviderConfig
      providers:
        - name: ecr-credential-provider
          matchImages:
            - "*.dkr.ecr.*.amazonaws.com"
            - "*.dkr.ecr.*.amazonaws.com.cn"
            - "*.dkr.ecr-fips.*.amazonaws.com"
            - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
            - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
          defaultCacheDuration: "12h"
          apiVersion: credentialprovider.kubelet.k8s.io/v1

4. Rebooted with talosctl reboot.
5. Applied a simple Pod configuration:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: {{ aws id }}.dkr.ecr.{{ region }}.amazonaws.com/test:latest
    ports:
    - containerPort: 80

6. Encountered the following error when describing the Pod:
   kubectl describe pod nginx

Failed to pull image "{{ aws id }}.dkr.ecr.{{ region }}.amazonaws.com/test:latest": failed to pull and unpack image "{{ aws id }}.dkr.ecr.{{ region }}.amazonaws.com/test:latest": failed to resolve reference "{{ aws id }}.dkr.ecr.{{ region }}.amazonaws.com/test:latest": pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials

My Guess:

It seems like the credential-provider plugin may be missing necessary authentication information when making requests through the API, causing the error. I'm unsure how to add AWS authentication credentials to my on-prem machine so that the plugin can operate correctly.

If anyone has insights on this issue or can suggest a solution, I’d really appreciate it! 🙏

[smira]

You need to look into the credential provider docs on how to supply AWS credentials.

This is an extension, so the only bug on our side might be if kubelet doesn't pick it up.

[vitamaxDH]

@smira Thank you. Simply adding env solved the issue.

Note
For those who will encounter the same issue, please add env under machine in the patch.yaml file

ecr-credential-provider-patch.yaml

---
machine:
  kubelet:
    credentialProviderConfig:
      apiVersion: kubelet.config.k8s.io/v1
      kind: CredentialProviderConfig
      providers:
        - name: ecr-credential-provider
          matchImages:
            - "*.dkr.ecr.*.amazonaws.com"
            - "*.dkr.ecr.*.amazonaws.com.cn"
            - "*.dkr.ecr-fips.*.amazonaws.com"
            - "*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
            - "*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
          defaultCacheDuration: "12h"
          apiVersion: credentialprovider.kubelet.k8s.io/v1
  env:
    AWS_ACCESS_KEY_ID: {{ your access key id }} // no base 64, plain string
    AWS_SECRET_ACCESS_KEY: {{ your secret access key }} // no base 64, plain string

[smira]

Just to add to this ^^ having access/secret key as environment variables is not the best idea security-wise.

I think proper way is to use this from AWS VM which has proper IAM to access registry without creds, not possible on-prem, not sure what a proper solution would be (using a different container registry with easier auth story?)