rsyslog pods are crashing because port 6789 is already in use

[mkemmerz]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hi, I am trying to get the awx-operator running on one of my Azure Kubernetes Services.

The rsyslog-pods of task and web are constantly crashing on startup with the message:

{"level":"info","ts":"2024-03-21T11:49:02Z","logger":"cmd","msg":"Version","Go Version":"go1.20.12","GOOS":"linux","GOARCH":"amd64","ansible-operator":"v1.34.0","commit":"d26c43bf94960d292152862a6685696be33190fb"}
{"level":"info","ts":"2024-03-21T11:49:02Z","logger":"cmd","msg":"Watching all namespaces"}
{"level":"error","ts":"2024-03-21T11:49:02Z","logger":"cmd","msg":"Failed to create a new manager.","error":"error listening on :6789: listen tcp :6789: bind: address already in use","stacktrace":"github.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.run\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:163\ngithub.com/operator-framework/ansible-operator-plugins/internal/cmd/ansible-operator/run.NewCmd.func1\n\tansible-operator-plugins/internal/cmd/ansible-operator/run/cmd.go:81\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:987\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1115\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.0/command.go:1039\nmain.main\n\tansible-operator-plugins/cmd/ansible-operator/main.go:40\nruntime.main\n\t/opt/hostedtoolcache/go/1.20.12/x64/src/runtime/proc.go:250"}

kustomization.yml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  # Find the latest tag here: https://github.com/ansible/awx-operator/releases
  - github.com/ansible/awx-operator/config/default?ref=2.12.2

# Set the image tags to match the git version from above
images:
  - name: quay.io/ansible/awx-operator:latest
    newName: artifactory.xxx.com/docker/ansible/awx-operator
    newTag: 2.12.2

  - name: gcr.io/kubebuilder/kube-rbac-proxy:v0.15.0
    newName: artifactory.xxx.com/docker/kubebuilder/kube-rbac-proxy
    newTag: v0.15.0

# Specify a custom namespace in which to install AWX
namespace: asdf

secretGenerator:
  - name: awx-postgres-configuration
    type: Opaque
    literals:
      - host=awx-postgres
      - port=5432
      - database=awx
      - username=awx
      - password=xxx
      - type=managed

  - name: awx-admin-password
    type: Opaque
    literals:
      - password=xxx

AWX Demo:

---
apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  no_log: False
  service_type: NodePort
  nodeport_port: 30080
  postgres_image: artifactory.xxx.com/quay.io-remote/sclorg/postgresql-15-c9s
  postgres_image_version: latest
  image: artifactory.xxx.com/docker/ansible/awx-operator
  image_version: 2.12.2
  redis_image: artifactory.xxx.com/docker/redis
  redis_image_version: "7"
  ee_images:
    - name: "AWX EE (latest)"
      image: "artifactory.xxx.com/quay.io-remote/ansible/awx-ee:latest"
    - name: "AWX EE ({{ _image_version }})"
      image: "artifactory.xxx.com/quay.io-remote/ansible/awx-ee:latest"
  control_plane_ee_image: artifactory.xxx.com/quay.io-remote/ansible/awx-ee:latest
  init_container_image: artifactory.xx.com/quay.io-remote/ansible/awx-ee
  init_container_image_version: latest
  extra_settings:
    - setting: "LOGGING['handlers']['console']['level']"
      value: "'INFO'"



Additionally: `kubectl create clusterrolebinding awx-cluster-admin --clusterrole=cluster-admin  --serviceaccount=asdf:awx-demo`

AWX Operator version

2.12.2

AWX version

2.12.2

Kubernetes platform

kubernetes, Azure Kubernetes Service

Kubernetes/Platform version

1.29.0

Modifications

yes

Steps to reproduce

See yamls.

Expected results

The rsyslog pods start correctly, the awx installation is successful.

Actual results

rsyslog of web and task fail on startup with the attached error message.

Additional information

Usage of Artifactory as mirror.

Operator Logs

No response

[mkemmerz]

Same issue as here: #828 (comment)

Changing the AWX image from awx-operator to awx solved the issue:

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
  name: awx-demo
spec:
  image: artifactory.xxx.com/quay.io-remote/ansible/awx
  image_version: "23.9.0"