Backup issue on AWX 23.8.1

[mxs-weixiong]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

I encounter the following error when try to run operator backup on AWX 23.8.1

level":"info","ts":"2024-02-20T08:32:04Z","msg":"Stopping and waiting for non leader election runnables"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"Stopping and waiting for leader election runnables"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxrestore-controller"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awx-controller"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"Shutdown signal received, waiting for all workers to finish","controller":"awxbackup-controller"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"All workers finished","controller":"awx-controller"} {"level":"info","ts":"2024-02-20T08:32:04Z","msg":"All workers finished","controller":"awxrestore-controller"} {"level":"info","ts":"2024-02-20T08:32:34Z","msg":"Stopping and waiting for caches"} {"level":"info","ts":"2024-02-20T08:32:34Z","msg":"Stopping and waiting for webhooks"} {"level":"info","ts":"2024-02-20T08:32:34Z","msg":"Wait completed, proceeding to shutdown the manager"} {"level":"error","ts":"2024-02-20T08:32:34Z","msg":"error received after stop sequence was engaged","error":"leader election lost","stacktrace":"sigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).engageStopProcedure.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/manager/internal.go:555"} {"level":"error","ts":"2024-02-20T08:32:34Z","logger":"cmd","msg":"Proxy or operator exited with error.","Namespace":"awx","error":"[failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced, failed waiting for all runnables to end within grace period of 30s: context deadline exceeded]","errorCauses":[{"error":"failed to wait for awxmeshingress-controller caches to sync: timed out waiting for cache to be synced"},{"error":"failed waiting for all runnables to end within grace period of 30s: context deadline exceeded"}],"stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.run\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:287\ngithub.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.NewCmd.func1\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:80\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:1044\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.6.1/command.go:968\nmain.main\n\tansible-operator-plugins/cmd/ansible-operator/main.go:40\nruntime.main\n\t/opt/hostedtoolcache/go/1.19.13/x64/src/runtime/proc.go:250"}

Anyone able to do a successful backup on AWX 23.8.1?
Or are you using a different method to backup.

Thanks.

AWX version

23.8.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Create a backup operator file.

---
apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
  name: backup-2024-02-20-23-8-1
  namespace: awx
spec:
  deployment_name: ansible-awx

Expected results

Backup success with all the files created in the selected directory

Actual results

Only tower.db file created.

secrets and awx_objects file are not created

Additional information

No response

[kurokobo]

@mxs-weixiong
Hi, have you installed AWX Operator via Helm in the past, and recently upgraded AWX Operator using helm upgrade, right?
If so, could you please try this: kubectl apply --server-side --force-conflicts -k github.com/ansible/awx-operator/config/crd?ref=2.12.1

[mxs-weixiong]

@kurokobo it works! Thanks.

I need to put into my notes always update the CRD after helm upgrade.

[kurokobo]

@mxs-weixiong
Hi, good to hear that you've solve the issue! I want to add some context for this issue for future references:

Context:
According to your logs, it seems that your Operator is shutting down since Operator cannot handle AWXMeshIngress CRD. helm upgrade does not apply any updates for CRDs, so the CRD AWXMeshIngress that added in 2.11.0 may not exist on your Kubernetes cluster. The command in my comment above is to forcibly apply CRD updates to your cluster.

See also:

• Known issues for Opearator 2.11.0: https://github.com/ansible/awx-operator/releases/tag/2.11.0
• Caveats on upgrading existing installation: https://github.com/ansible/awx-operator/tree/devel/.helm/starter#caveats-on-upgrading-existing-installation
• Related issues: no matches for AWXMeshIngress upgrading to 2.11.0 awx-operator#1701, Proxy or operator exited with error after upgrade from 23.6.0 to 23.7.0 #14888