Test(e2e): re-enable upgrade test from nightly image

[randmonkey]

What this PR does / why we need it:
As nightly images can be pushed again, we re-enable the e2e tests to upgrade from nightly.
Which issue this PR fixes

Fixes #

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect significant changes

[pmalek]

This will require some more changes as now nightlies are being pushed to a separate repository: https://hub.docker.com/r/kong/nightly-gateway-operator-oss/tags

(for EE that's https://hub.docker.com/r/kong/nightly-gateway-operator/tags)

So we'll need to:

• not make the repo hardcoded and allow override in EE repo
• allow overriding the repository in testcases in this test.

[randmonkey]

I found such reconciler errors in diagnostics:

{
  "level": "error",
  "ts": "2025-01-21T07:23:15Z",
  "msg": "Reconciler error",
  "controller": "controlplane",
  "controllerGroup": "gateway-operator.konghq.com",
  "controllerKind": "ControlPlane",
  "ControlPlane": {
    "name": "gw-upgrade-nightly-to-current-fnvw2-tmszg",
    "namespace": "7d44ebc6-36f3-47e1-a552-f1199ebef89d"
  },
  "namespace": "7d44ebc6-36f3-47e1-a552-f1199ebef89d",
  "name": "gw-upgrade-nightly-to-current-fnvw2-tmszg",
  "reconcileID": "c56445c2-e368-42fb-9b84-4d7ef81bf466",
  "error": "clusterroles.rbac.authorization.k8s.io is forbidden: user \"system:serviceaccount:7d44ebc6-36f3-47e1-a552-f1199ebef89d:controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:7d44ebc6-36f3-47e1-a552-f1199ebef89d\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"gateway.networking.k8s.io\"], Resources:[\"backendtlspolicies\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"gateway.networking.k8s.io\"], Resources:[\"backendtlspolicies/status\"], Verbs:[\"patch\" \"update\"]}",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:332\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240"
}

Looks like we need to update the helm chart to grant the permissions of KGO to create cluster roles required for KIC 3.4 to operate backendtlspolicies.

[pmalek]

  "error": "clusterroles.rbac.authorization.k8s.io is forbidden: user \"system:serviceaccount:7d44ebc6-36f3-47e1-a552-f1199ebef89d:controller-manager\" (groups=[\"system:serviceaccounts\" \"system:serviceaccounts:7d44ebc6-36f3-47e1-a552-f1199ebef89d\" \"system:authenticated\"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:[\"gateway.networking.k8s.io\"], Resources:[\"backendtlspolicies\"], Verbs:[\"get\" \"list\" \"watch\"]}\n{APIGroups:[\"gateway.networking.k8s.io\"], Resources:[\"backendtlspolicies/status\"], Verbs:[\"patch\" \"update\"]}",

BackendTLSPolicy RBAC rules have already been added in Kong/charts#1191 so this should work. What might be necessary here is to add the effective version for the nightly given that these RBAC rules are conditional on the version of KIC's image: https://github.com/Kong/charts/blob/94c153ca77d087ee2bba94686574dce588f204a5/charts/kong/templates/_helpers.tpl#L1307

[randmonkey]

"error": "clusterroles.rbac.authorization.k8s.io is forbidden: user "system:serviceaccount:7d44ebc6-36f3-47e1-a552-f1199ebef89d:controller-manager" (groups=["system:serviceaccounts" "system:serviceaccounts:7d44ebc6-36f3-47e1-a552-f1199ebef89d" "system:authenticated"]) is attempting to grant RBAC permissions not currently held:\n{APIGroups:["gateway.networking.k8s.io"], Resources:["backendtlspolicies"], Verbs:["get" "list" "watch"]}\n{APIGroups:["gateway.networking.k8s.io"], Resources:["backendtlspolicies/status"], Verbs:["patch" "update"]}",

BackendTLSPolicy RBAC rules have already been added in Kong/charts#1191 so this should work. What might be necessary here is to add the effective version for the nightly given that these RBAC rules are conditional on the version of KIC's image: https://github.com/Kong/charts/blob/94c153ca77d087ee2bba94686574dce588f204a5/charts/kong/templates/_helpers.tpl#L1307

The error logs shows that KGO does not have the permission to grant the permission of operating backendtlspolicies to the clusterrole used by KIC. So I think it is caused by missing the permission in KGO's RBAC resources. Kong/charts#1230 should fix it.