Fix flaky upgrade test

[lrascao]

Description

There's a flake in the upgrade test from 1.11.0 to 1.12.0, this is a version where we upgrade the configuration CRD by adding two new fields:controlPlaneTrustDomain and sentryAddress.
The flake stems from a known race condition that arises when both updating a CRD and a matching CR close together (described here). This patch introduces a basic retry mechanism, it needs a fresh client on each try to the OpenAPI schema caching that's happening in the kubectl client.

Issue reference

No issue referenced so far but a few occurrences of the flake can be seen in the GHA history

The error is typically always the same:

        {"time":"2023-11-26T12:19:15.977671678Z","status":"info","msg":"Dapr control plane version 1.11.0 detected in namespace dapr-cli-tests"}
        {"time":"2023-11-26T12:19:16.073032862Z","status":"info","msg":"Starting upgrade..."}
        {"time":"2023-11-26T12:19:17.416345982Z","status":"failure","msg":"Failed to upgrade Dapr: error validating \"\": error validating data: [ValidationError(Configuration.spec.mtls): unknown field \"controlPlaneTrustDomain\" in io.dapr.v1alpha1.Configuration.spec.mtls, ValidationError(Configuration.spec.mtls): unknown field \"sentryAddress\" in io.dapr.v1alpha1.Configuration.spec.mtls]"}

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• [] Extended the documentation

[codecov]

Codecov Report

Attention: 58 lines in your changes are missing coverage. Please review.

Comparison is base (f5eb4fd) 22.76% compared to head (00a5cac) 22.55%.

❗ Current head 00a5cac differs from pull request most recent head 8d81dc1. Consider uploading reports for the commit 8d81dc1 to get more accurate results

Files	Patch %	Lines
pkg/kubernetes/upgrade.go	0.00%	58 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1368      +/-   ##
==========================================
- Coverage   22.76%   22.55%   -0.22%     
==========================================
  Files          40       40              
  Lines        4713     4758      +45     
==========================================
  Hits         1073     1073              
- Misses       3562     3607      +45     
  Partials       78       78              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mukundansundar]

@lrascao
Thanks for the contribution.
Please change documentation in readme that the upgrade client now retries 5 times on failure.

[lrascao]

@lrascao Thanks for the contribution. Please change documentation in readme that the upgrade client now retries 5 times on failure.

done, added a note to the README. lmk if something else is needed

[lrascao]

@mukundansundar looks like the dapr_cli CI failure is due to the upload artifact using master (actions/upload-artifact@master), v4 was released that changed the upload semantics. Pinning it down to v3 is probably the faster fix

[mukundansundar]

@mukundansundar looks like the dapr_cli CI failure is due to the upload artifact using master (actions/upload-artifact@master), v4 was released that changed the upload semantics. Pinning it down to v3 is probably the faster fix

Agreed ...

[dapr-bot]

This pull request has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!