Missing Toleration for Migration pod.

[Jaryllan]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

All other pods has tolerations except migration job missing tolerations and nodeselector.

AWX version

24.0.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

core 2.14.6

Operating system

Ubuntu Server 22.04

Web browser

No response

Steps to reproduce

awx created with the following configuration. All other pods except migration can define the tolerations.

apiVersion: awx.ansible.com/v1beta1
kind: AWX
metadata:
name: awx
namespace: awx
spec:
ingress_class_name: nginx
ingress_path: /awx
ingress_type: ingress
node_selector: |
node-role.kubernetes.io/control-plane: ""
postgres_selector: |
node-role.kubernetes.io/control-plane: ""
postgres_tolerations: |
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
task_node_selector: |
node-role.kubernetes.io/control-plane: ""
task_tolerations: |
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
tolerations: |
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists
web_node_selector: |
node-role.kubernetes.io/control-plane: ""
web_tolerations: |
- effect: NoSchedule
key: node-role.kubernetes.io/control-plane
operator: Exists

Expected results

Migration pod can run to completion status and task pod can be in running state.

Actual results

MIgration pod stuck in pending status and task pod stuck in init status.

Additional information

No response

[ranvit]

I was running AWX in a kube cluster where every node has a taint, so every object needs a toleration in order to get scheduled.

This migration job was in a pending state because it never got scheduled onto a node. Due to this, I was seeing all the same errors like relation "conf_setting" does not exist(#568) or relation "django_migrations" does not exist(#1610)

I kept killing and restarting the operator, postgres, web pods, and I never noticed that the migration-job-pod was not starting up.

Once I realized the migration pod needs a toleration, I added it via kubectl edit <migration pod> and the pod was able to get scheduled, migrations ran, all errors resolved in the postgres and web pods

Longterm - I cant patch this job with kustomize because it seems to get templated by the operator. So we need a PR that adds the postgres_tolerations and postgres_node_selector to the migration job template