CAPI installation failures in CI due to errors on accessing VPC resources

[Shilpa-Gokul]

/kind bug
/area provider/ibmcloud

What steps did you take and what happened:
Initiated 4.19 CAPI PowerVS run in CI multiple times and the installation was not successful. Each time it failed with a different error. Below are the error logs.

Loglink

level=fatal msg=failed to fetch Cluster API Manifests: failed to generate asset "Cluster API Manifests": failed to generate PowerVS manifests error getting subnets in specified VPC: vpc-lon06-powervs-7-0 failed to list vpc regions: The system is currently unable to fulfill the request. Please retry. If the failure persists, contact support and provide the trace value from this error.

Log2link

time="2024-12-09T08:30:16Z" level=error msg="failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: failed during post-machine creation hook: failed to set the VPC service region (eu-gb) in PostProvision: The system is currently unable to fulfill the request. Please retry. If the failure persists, contact support and provide the trace value from this error."

Log3link

time="2024-12-09T07:41:08Z" level=debug msg="Collecting applied cluster api manifests..."
time="2024-12-09T07:41:08Z" level=error msg="failed to fetch Cluster: failed to generate asset "Cluster": failed to create cluster: infrastructure was not ready within 30m0s: client rate limiter Wait returned an error: context deadline exceeded"

Few other job links with similar error
https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.18-ocp-e2e-ovn-powervs-capi-multi-p-p/1866045227924459520/artifacts/ocp-e2e-ovn-powervs-capi-multi-p-p/ipi-install-powervs-install/build-log.txt
https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/58927/rehearse-58927-pull-ci-openshift-cluster-api-provider-ibmcloud-release-4.19-e2e-powervs-capi-techpreview/1866352581316448256
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-multiarch-master-nightly-4.18-ocp-e2e-ovn-powervs-capi-multi-p-p/1866226439100043264

What did you expect to happen:

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version:
• Minikube/KIND version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[Karthik-K-N]

@Shilpa-Gokul is this issue still present?

[Shilpa-Gokul]

@Karthik-K-N First two issues are not observed, but the below issue is seen even in recent runs.

failed to create cluster: infrastructure was not ready within 30m0s: client rate limiter Wait returned an error: context deadline exceeded

[Karthik-K-N]

@Karthik-K-N First two issues are not observed, but the below issue is seen even in recent runs.

failed to create cluster: infrastructure was not ready within 30m0s: client rate limiter Wait returned an error: context deadline exceeded

I think the error is too generic and not sure from where its originating, May be we need to debug more to precisely tell that its from CAPI.

[Shilpa-Gokul]

@Karthik-K-N The issue is seen while trying to create DHCP server which eventually leads to the context deadline exceeded error as shown below,

- type: NetworkReady
  status: "False"
  severity: Error
  lasttransitiontime: "2025-01-14T23:46:25Z"
  reason: NetworkReconciliationFailed
  message: 'failed to perform Create DHCP Operation for cloud instance XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with error [POST /pcloud/v1/cloud-instances/{cloud_instance_id}/services/dhcp][500] pcloudDhcpPostInternalServerError {"description":"dhcp server failed to create private network: unable to retrieve active status for new PER connection information after create private network f6cdbae2-7d37-4c2a-8282-be04fc5b7e85: %!!(MISSING)w(\u003cnil\u003e)","error":"internal server error"}' 

Have raised an IBM JIRA for the same- https://jsw.ibm.com/browse/POWERIAAS-16982

[Shilpa-Gokul]

@Karthik-K-N Looks like this is a platform side issue and not CAPI specific. We also have a workaround for the DHCP issue now, hence closing this.

[Karthik-K-N]

@Karthik-K-N Looks like this is a platform side issue and not CAPI specific. We also have a workaround for the DHCP issue now, hence closing this.

Thank you.