ERROR [-] awx.main.tasks.jobs project_update xxx (running) Post run hook errored. (OSError: [Errno 39])

[acsezen]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that the AWX Operator is open source software provided for free and that I might not receive a timely response.

Bug Summary

Hello team,

Occasionally, I encounter a Post run hook errored for project updates, even though most projects update successfully. Here are the tracebacks:

2025-02-19 03:01:00,357 ERROR    [-] awx.main.tasks.jobs project_update xxx (running) Post run hook errored.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 637, in run
    self.post_run_hook(self.instance, status)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 1410, in post_run_hook
    shutil.rmtree(stage_path)  # cannot trust content update produced
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/shutil.py", line 752, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 683, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib64/python3.11/shutil.py", line 681, in _rmtree_safe_fd
    os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'ansible-windows-1.11.1-cgel_146

or

2025-02-19 09:00:41,404 ERROR    [ad623158921548589ea54af63b05b196] awx.main.tasks.jobs project_update xxx (running) Post run hook errored.
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 637, in run
    self.post_run_hook(self.instance, status)
  File "/var/lib/awx/venv/awx/lib64/python3.11/site-packages/awx/main/tasks/jobs.py", line 1410, in post_run_hook
    shutil.rmtree(stage_path)  # cannot trust content update produced
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/shutil.py", line 752, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 672, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib64/python3.11/shutil.py", line 703, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib64/python3.11/shutil.py", line 701, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'tmpvjbzpmia'

AWX Operator version

2.19.1

AWX version

24.6.1

Kubernetes platform

openshift

Kubernetes/Platform version

Kustomize Version: v5.0.1 Server Version: 4.14.34 Kubernetes Version: v1.27.15+6147456

Modifications

no

Steps to reproduce

It happens randomly.

Expected results

Project_update run successfully always.

Actual results

Project_update fails randomly.

Additional information

% oc rsh -n awx-production deployments/awx-prd-task ls -al /var/lib/awx
total 44
drwxrwxr-x. 1 root root 43 Feb 19 13:00 .
drwxr-xr-x. 1 root root 33 Feb 19 00:49 ..
-rw-------. 1 awx root 177 Feb 19 14:10 .bash_history
drwxr-xr-x. 3 root root 19 Jul 2 2024 .local
drwxrwxrwx. 160 99 99 36864 Jan 31 12:58 projects
drwxr-xr-x. 3 root root 20 Jul 2 2024 public
drwxrwxr-x. 1 root root 40 Jul 2 2024 rsyslog
drwxr-xr-x. 3 root root 17 Jul 2 2024 venv

oc rsh -n awx-production deployments/awx-prd-task ls -al /var/lib/awx/projects/ | head -n10
total 688
drwxrwxrwx. 160   99   99 36864 Jan 31 12:58 .
drwxrwxr-x.   1 root root    43 Feb 19 13:00 ..
drwxr-xr-x.   5 awx    99  4096 Nov 14 09:33 _517__xxx_xxx_xxx
-rwxr-xr-x.   1 awx    99     0 Nov 14 09:33 _517__xxx_xxx_xxx.lock

% oc rsh -n awx-production deployments/awx-prd-web ls -al /var/lib/awx
total 40
drwxrwxr-x. 1 root root 37 Feb 18 23:52 .
drwxr-xr-x. 1 root root 30 Jul 2 2024 ..
prw-------. 1 awx root 0 Feb 18 23:52 awxfifo
drwxr-xr-x. 3 root root 19 Jul 2 2024 .local
drwxrwxrwx. 160 99 99 36864 Jan 31 12:58 projects
drwxr-xr-x. 3 root root 20 Jul 2 2024 public
drwxrwxr-x. 1 root root 40 Jul 2 2024 rsyslog
drwxr-xr-x. 3 root root 17 Jul 2 2024 venv

% oc rsh -n awx-production deployments/awx-prd-task cat /etc/passwd
awx:x:1000940000:0:,,,:/var/lib/awx:/bin/bash

% oc rsh -n awx-production deployments/awx-prd-task cat /etc/group
root:x:0:
bin:x:1:
daemon:x:2:
sys:x:3:
adm:x:4:
tty:x:5:
disk:x:6:
lp:x:7:
mem:x:8:
kmem:x:9:
wheel:x:10:
cdrom:x:11:
mail:x:12:
man:x:15:
dialout:x:18:
floppy:x:19:
games:x:20:
tape:x:33:
video:x:39:
ftp:x:50:
lock:x:54:
audio:x:63:
users:x:100:
nobody:x:65534:
tss:x:59:
nginx:x:999:
utmp:x:22:
utempter:x:35:
ssh_keys:x:101:
input:x:998:
kvm:x:36:
render:x:997:
systemd-journal:x:190:
systemd-coredump:x:996:
dbus:x:81:

Operator Logs

No response