Calico API Server Crashing on EKS Cluster version 1.30 Node running with Ubuntu 22.04

[evyatar-toledano-cobwebs]

Expected Behavior

The Calico API server version 3.29.1 with default configuration as network policy:
Working in EKS 1.29 with Ubuntu 20.04
Expected to work also with EKS 1.30 with Ubuntu 22.04

Current Behavior

The Calico API server (3.29.1) runs on EKS 1.30 with Ubuntu 22.04 for several hours but eventually encounters a permission error:
[webhook.go:253] Failed to make webhook authorizer request.

The calico was installed using helm <helm install calico projectcalico/tigera-operator --namespace tigera-operator>
using default Calico configuration values for the "Policy network only" setup.

Context

The application loses external internet connectivity when the Calico API server crashes and encounters this error.
The issue persists until all pods in the cluster are redeployed to a new node.

Notably, this issue does not occur with:

AWS EKS version 1.29 (Ubuntu 20.04).
The issue appears to be specific to nodes running Ubuntu 22.04.

Your Environment

• Calico version 3.29.1
• Orchestrator version AWS EKS version 1.30 / 1.29
• Operating System and version: Ubuntu 22.04

Additional Notes

The issue seems to be related to the operating system version (Ubuntu 22.04) when running EKS 1.30.
It does not manifest with Ubuntu 20.04 on either EKS 1.29. Based on testing and research, the problem consistently occurs on nodes using Ubuntu 22.04.

Got the following errors:

[webhook.go:253] Failed to make webhook authorizer request: Post "https://172.20.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": dial tcp 172.20.0.1:443: socket: operation not permitted
[errors.go:77] Post "https://172.20.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": dial tcp 172.20.0.1:443: socket: operation not permitted
[reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.10/tools/cache/reflector.go:229: Failed to watch *v1.ValidatingWebhookConfiguration: Get "https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?allowWatchBookmarks=true&resourceVersion=67881749&timeout=7m3s&timeoutSeconds=423&watch=true": dial tcp 172.20.0.1:443: socket: operation not permitted
[reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.10/tools/cache/reflector.go:229: failed to list *v1.ValidatingWebhookConfiguration: Get "https://172.20.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=67881749": dial tcp 172.20.0.1:443: socket: operation not permitted

[caseydavenport]

I can't really see how this could be related to the node image version being used, at least so far as the authorizationreview errors are concerned!

I believe @coutinhop is looking into a similar symptom at the moment - wonder if they are related.

[caseydavenport]

I suspect this has to do with EKS v1.29 vs. EKS v1.30 - has something changed with the authorizationreview API in that upgrade?

[evyatar-toledano-cobwebs]

Authorization review API

No, this upgrade does not change the authorization API or the Calico add-on configuration.

Also, the Calico API server API worked well on the EKS cluster v1.30 but crashed on the error above after 9-12 hours and back to work if I deployed on a new node.