gateway_users role not idempotent, rerun of configuration fails

[hobby65]

Summary

When configuring gateway from a user_accounts.yaml file, the first time it runs without any problem.
But when the same file is used to add a user, the playbook fails on exsisting users, it seems that the module is
missing idempotency.

Issue Type

• Bug Report

Ansible, Collection, Controller details

ansible --version
ansible [core 2.15.10]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/wilco/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/wilco/.local/lib/python3.9/site-packages/ansible
  ansible collection location = /home/wilco/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/wilco/.local/bin/ansible
  python version = 3.9.13 (main, Jul 25 2022, 00:00:00) [GCC 11.3.1 20220421 (Red Hat 11.3.1-2)] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True
[wilco@rhel9-1 ee_cac_image (update)]$ ansible-galaxy collection list

# /home/wilco/.ansible/collections/ansible_collections
Collection                          Version
----------------------------------- -------
ansible.controller                  4.6.3  
ansible.eda                         2.2.0  
ansible.hub                         1.0.0  
ansible.netcommon                   4.1.0  
ansible.platform                    2.5.3  
ansible.posix                       2.0.0  
ansible.utils                       2.8.0  
ansible.windows                     2.6.0  
awx.awx                             22.1.0 
community.crypto                    2.10.0 
community.general                   10.1.0 
community.postgresql                2.3.2  
community.vmware                    5.2.0  
community.windows                   2.3.0  
infra.aap_configuration             3.1.0  
infra.ah_configuration              2.0.4  
redhat.satellite                    4.0.0  
redhat_cop.controller_configuration 2.3.1  
vmware.vmware                       1.7.1  
wf_linux.infra                      0.0.7  
wf_linux.oracle                     0.0.1  
wf_linux.rhel                       0.0.2  
wf_linux.web                        0.0.4  

# /home/wilco/.local/lib/python3.9/site-packages/ansible_collections
Collection                          Version
----------------------------------- -------
amazon.aws                          6.5.0  
ansible.netcommon                   5.3.0  
ansible.posix                       1.5.4  
ansible.utils                       2.12.0 
ansible.windows                     1.14.0 
arista.eos                          6.2.2  
awx.awx                             22.7.0 
azure.azcollection                  1.19.0 
check_point.mgmt                    5.1.1  
chocolatey.chocolatey               1.5.1  
cisco.aci                           2.8.0  
cisco.asa                           4.0.3  
cisco.dnac                          6.9.0  
cisco.intersight                    1.0.27 
cisco.ios                           4.6.1  
cisco.iosxr                         5.0.3  
cisco.ise                           2.6.2  
cisco.meraki                        2.17.0 
cisco.mso                           2.5.0  
cisco.nso                           1.0.3  
cisco.nxos                          4.4.0  
cisco.ucs                           1.10.0 
cloud.common                        2.1.4  
cloudscale_ch.cloud                 2.3.1  
community.aws                       6.4.0  
community.azure                     2.0.0  
community.ciscosmb                  1.0.7  
community.crypto                    2.16.1 
community.digitalocean              1.24.0 
community.dns                       2.6.4  
community.docker                    3.4.11 
community.fortios                   1.0.0  
community.general                   7.5.2  
community.google                    1.0.0  
community.grafana                   1.6.1  
community.hashi_vault               5.0.1  
community.hrobot                    1.8.2  
community.libvirt                   1.3.0  
community.mongodb                   1.6.3  
community.mysql                     3.8.0  
community.network                   5.0.2  
community.okd                       2.3.0  
community.postgresql                2.4.3  
community.proxysql                  1.5.1  
community.rabbitmq                  1.2.3  
community.routeros                  2.11.0 
community.sap                       1.0.0  
community.sap_libs                  1.4.1  
community.skydive                   1.0.0  
community.sops                      1.6.7  
community.vmware                    3.11.1 
community.windows                   1.13.0 
community.zabbix                    2.2.0  
containers.podman                   1.11.0 
cyberark.conjur                     1.2.2  
cyberark.pas                        1.0.23 
dellemc.enterprise_sonic            2.2.0  
dellemc.openmanage                  7.6.1  
dellemc.powerflex                   1.9.0  
dellemc.unity                       1.7.1  
f5networks.f5_modules               1.27.1 
fortinet.fortimanager               2.3.0  
fortinet.fortios                    2.3.4  
frr.frr                             2.0.2  
gluster.gluster                     1.0.2  
google.cloud                        1.3.0  
grafana.grafana                     2.2.3  
hetzner.hcloud                      1.16.0 
hpe.nimble                          1.1.4  
ibm.qradar                          2.1.0  
ibm.spectrum_virtualize             1.12.0 
ibm.storage_virtualize              2.1.0  
infinidat.infinibox                 1.3.12 
infoblox.nios_modules               1.5.0  
inspur.ispim                        1.3.0  
inspur.sm                           2.3.0  
junipernetworks.junos               5.3.1  
kubernetes.core                     2.4.0  
lowlydba.sqlserver                  2.2.2  
microsoft.ad                        1.4.1  
netapp.aws                          21.7.1 
netapp.azure                        21.10.1
netapp.cloudmanager                 21.22.1
netapp.elementsw                    21.7.0 
netapp.ontap                        22.8.3 
netapp.storagegrid                  21.11.1
netapp.um_info                      21.8.1 
netapp_eseries.santricity           1.4.0  
netbox.netbox                       3.15.0 
ngine_io.cloudstack                 2.3.0  
ngine_io.exoscale                   1.1.0  
ngine_io.vultr                      1.1.3  
openstack.cloud                     2.2.0  
openvswitch.openvswitch             2.1.1  
ovirt.ovirt                         3.2.0  
purestorage.flasharray              1.24.0 
purestorage.flashblade              1.14.0 
purestorage.fusion                  1.6.0  
sensu.sensu_go                      1.14.0 
servicenow.servicenow               1.0.6  
splunk.es                           2.1.2  
t_systems_mms.icinga_director       1.33.1 
telekom_mms.icinga_director         1.35.0 
theforeman.foreman                  3.15.0 
vmware.vmware_rest                  2.3.1  
vultr.cloud                         1.11.0 
vyos.vyos                           4.1.0  
wti.remote                          1.0.5  

# /usr/local/lib/python3.9/site-packages/ansible_collections
Collection                          Version
----------------------------------- -------
amazon.aws                          6.5.0  
ansible.netcommon                   5.3.0  
ansible.posix                       1.5.4  
ansible.utils                       2.12.0 
ansible.windows                     1.14.0 
arista.eos                          6.2.2  
awx.awx                             22.7.0 
azure.azcollection                  1.19.0 
check_point.mgmt                    5.1.1  
chocolatey.chocolatey               1.5.1  
cisco.aci                           2.8.0  
cisco.asa                           4.0.3  
cisco.dnac                          6.9.0  
cisco.intersight                    1.0.27 
cisco.ios                           4.6.1  
cisco.iosxr                         5.0.3  
cisco.ise                           2.6.2  
cisco.meraki                        2.17.0 
cisco.mso                           2.5.0  
cisco.nso                           1.0.3  
cisco.nxos                          4.4.0  
cisco.ucs                           1.10.0 
cloud.common                        2.1.4  
cloudscale_ch.cloud                 2.3.1  
community.aws                       6.4.0  
community.azure                     2.0.0  
community.ciscosmb                  1.0.7  
community.crypto                    2.16.1 
community.digitalocean              1.24.0 
community.dns                       2.6.4  
community.docker                    3.4.11 
community.fortios                   1.0.0  
community.general                   7.5.2  
community.google                    1.0.0  
community.grafana                   1.6.1  
community.hashi_vault               5.0.1  
community.hrobot                    1.8.2  
community.libvirt                   1.3.0  
community.mongodb                   1.6.3  
community.mysql                     3.8.0  
community.network                   5.0.2  
community.okd                       2.3.0  
community.postgresql                2.4.3  
community.proxysql                  1.5.1  
community.rabbitmq                  1.2.3  
community.routeros                  2.11.0 
community.sap                       1.0.0  
community.sap_libs                  1.4.1  
community.skydive                   1.0.0  
community.sops                      1.6.7  
community.vmware                    3.11.1 
community.windows                   1.13.0 
community.zabbix                    2.2.0  
containers.podman                   1.11.0 
cyberark.conjur                     1.2.2  
cyberark.pas                        1.0.23 
dellemc.enterprise_sonic            2.2.0  
dellemc.openmanage                  7.6.1  
dellemc.powerflex                   1.9.0  
dellemc.unity                       1.7.1  
f5networks.f5_modules               1.27.1 
fortinet.fortimanager               2.3.0  
fortinet.fortios                    2.3.4  
frr.frr                             2.0.2  
gluster.gluster                     1.0.2  
google.cloud                        1.3.0  
grafana.grafana                     2.2.3  
hetzner.hcloud                      1.16.0 
hpe.nimble                          1.1.4  
ibm.qradar                          2.1.0  
ibm.spectrum_virtualize             1.12.0 
ibm.storage_virtualize              2.1.0  
infinidat.infinibox                 1.3.12 
infoblox.nios_modules               1.5.0  
inspur.ispim                        1.3.0  
inspur.sm                           2.3.0  
junipernetworks.junos               5.3.1  
kubernetes.core                     2.4.0  
lowlydba.sqlserver                  2.2.2  
microsoft.ad                        1.4.1  
netapp.aws                          21.7.1 
netapp.azure                        21.10.1
netapp.cloudmanager                 21.22.1
netapp.elementsw                    21.7.0 
netapp.ontap                        22.8.3 
netapp.storagegrid                  21.11.1
netapp.um_info                      21.8.1 
netapp_eseries.santricity           1.4.0  
netbox.netbox                       3.15.0 
ngine_io.cloudstack                 2.3.0  
ngine_io.exoscale                   1.1.0  
ngine_io.vultr                      1.1.3  
openstack.cloud                     2.2.0  
openvswitch.openvswitch             2.1.1  
ovirt.ovirt                         3.2.0  
purestorage.flasharray              1.24.0 
purestorage.flashblade              1.14.0 
purestorage.fusion                  1.6.0  
sensu.sensu_go                      1.14.0 
servicenow.servicenow               1.0.6  
splunk.es                           2.1.2  
t_systems_mms.icinga_director       1.33.1 
telekom_mms.icinga_director         1.35.0 
theforeman.foreman                  3.15.0 
vmware.vmware_rest                  2.3.1  
vultr.cloud                         1.11.0 
vyos.vyos                           4.1.0  
wti.remote                          1.0.5  

Automation Controller Version
4.6.3
Event Driven Automation Version
1.1.2
Automation Hub Version
4.10.1

• ansible installation method: one of source, pip, OS package, EE

OS / ENVIRONMENT

Desired Behavior

The playbook/role should only add the new user and report [ok] when the esisting users are not changed.

Actual Behavior

Please give some details of what is actually happening.
Include a [minimum complete verifiable example] with:

• playbook / task

---
- name: Configure rhaap platform base
  hosts: aapserver3.localdomain
  connection: local
  gather_facts: false

  - pre_tasks:
      - name: include the user_vars
        ansible.builtin.include_vars:
          file: user_accounts.yml

  roles:
    - infra.aap_configuration.gateway_users

• configuration file: user_accounts.yml

---
aap_user_accounts:
  - username: mgt
    password: some_pass
    email:
    first_name: user
    last_name: platform management
    # authenticators: Local
    update_secrets: false

  - username: coll_upload
    password: soma_pass
    email:
    first_name: collection
    last_name: upload
    # authenticators: Local
    update_secrets: false

  - username: coll_get
    password: some_pass
    email:
    first_name: collection
    last_name: download
    # authenticators: Local
    update_secrets: false

  - username: ee_upload
    password: some_pass
    email:
    first_name: ee
    last_name: upload
    # authenticators: Local
    update_secrets: false

• error

<aap_dev> ESTABLISH LOCAL CONNECTION FOR USER: wilco
<aap_dev> EXEC /bin/sh -c 'echo ~wilco && sleep 0'
<aap_dev> EXEC /bin/sh -c 'echo ~wilco && sleep 0'
<aap_dev> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/wilco/.ansible/tmp `"&& mkdir "` echo /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468 `" && echo ansible-tmp-1734516996.791021-7969-44740564126468="` echo /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468 `" ) && sleep 0'
Using module file /home/wilco/.local/lib/python3.9/site-packages/ansible/modules/async_status.py
<aap_dev> PUT /home/wilco/.ansible/tmp/ansible-local-71397_aaod7_/tmpxuonher3 TO /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468/AnsiballZ_async_status.py
<aap_dev> EXEC /bin/sh -c 'chmod u+x /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468/ /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468/AnsiballZ_async_status.py && sleep 0'
<aap_dev> EXEC /bin/sh -c '/usr/bin/python3 /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468/AnsiballZ_async_status.py && sleep 0'
<aap_dev> EXEC /bin/sh -c 'rm -f -r /home/wilco/.ansible/tmp/ansible-tmp-1734516996.791021-7969-44740564126468/ > /dev/null 2>&1 && sleep 0'
failed: [aap_dev] (item=Create/Update Gateway user mgt | Wait for finish the Gateway user creation) => {
    "__gateway_user_accounts_job_async_results_item": {
        "__gateway_user_accounts_item": {
            "email": null,
            "first_name": "user",
            "last_name": "platform management",
            "password": "some_pass",
            "update_secrets": false,
            "username": "mgt"
        },
        "ansible_job_id": "j514521925947.7742",
        "ansible_loop_var": "__gateway_user_accounts_item",
        "changed": false,
        "failed": 0,
        "finished": 0,
        "results_file": "/home/wilco/.ansible_async/j514521925947.7742",
        "started": 1
    },
    "ansible_job_id": "j514521925947.7742",
    "ansible_loop_var": "__gateway_user_accounts_job_async_results_item",
    "attempts": 1,
    "changed": false,
    "cmd": "/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py",
    "data": "",
    "finished": 1,
    "invocation": {
        "module_args": {
            "_async_dir": "/home/wilco/.ansible_async",
            "jid": "j514521925947.7742",
            "mode": "status"
        }
    },
    "msg": "Traceback (most recent call last):\n  File \"/tmp/ansible_ansible.legacy.async_wrapper_payload_atzk61fz/ansible_ansible.legacy.async_wrapper_payload.zip/ansible/modules/async_wrapper.py\", line 180, in _run_module\n  File \"/tmp/ansible_ansible.legacy.async_wrapper_payload_atzk61fz/ansible_ansible.legacy.async_wrapper_payload.zip/ansible/modules/async_wrapper.py\", line 98, in _filter_non_json_lines\nValueError: No start of json char found\n",
    "results_file": "/home/wilco/.ansible_async/j514521925947.7742",
    "started": 1,
    "stderr": "Traceback (most recent call last):\n  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.ansible.platform.plugins.modules.user', init_globals=dict(_module_fqn='ansible_collections.ansible.platform.plugins.modules.user', _modlib_path=modlib_path),\n  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/modules/user.py\", line 125, in <module>\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/modules/user.py\", line 121, in main\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_object.py\", line 66, in manage\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 407, in create_or_update_if_needed\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 518, in update_if_needed\n  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 710, in objects_could_be_different\nAttributeError: 'AAPModule' object has no attribute 'fields_could_be_same'\n",
    "stderr_lines": [
        "Traceback (most recent call last):",
        "  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 107, in <module>",
        "    _ansiballz_main()",
        "  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 99, in _ansiballz_main",
        "    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)",
        "  File \"/home/wilco/.ansible/tmp/ansible-tmp-1734516991.822762-7729-117570209327651/AnsiballZ_user.py\", line 47, in invoke_module",
        "    runpy.run_module(mod_name='ansible_collections.ansible.platform.plugins.modules.user', init_globals=dict(_module_fqn='ansible_collections.ansible.platform.plugins.modules.user', _modlib_path=modlib_path),",
        "  File \"/usr/lib64/python3.9/runpy.py\", line 225, in run_module",
        "    return _run_module_code(code, init_globals, run_name, mod_spec)",
        "  File \"/usr/lib64/python3.9/runpy.py\", line 97, in _run_module_code",
        "    _run_code(code, mod_globals, init_globals,",
        "  File \"/usr/lib64/python3.9/runpy.py\", line 87, in _run_code",
        "    exec(code, run_globals)",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/modules/user.py\", line 125, in <module>",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/modules/user.py\", line 121, in main",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_object.py\", line 66, in manage",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 407, in create_or_update_if_needed",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 518, in update_if_needed",
        "  File \"/tmp/ansible_ansible.platform.user_payload_k6eqfhha/ansible_ansible.platform.user_payload.zip/ansible_collections/ansible/platform/plugins/module_utils/aap_module.py\", line 710, in objects_could_be_different",
        "AttributeError: 'AAPModule' object has no attribute 'fields_could_be_same'"
    ],
    "stdout": "",
    "stdout_lines": []
}

STEPS TO REPRODUCE

create the playbook and the configuration file as shown in this bug report and run this configuration twice against the same installation and it will give the error shown.

The playbook used in the above bug desciption will do the trick

[bogdanmuresan]

I had the same issue and have opened a support ticket to Red Hat for this on November 13th, and I was told the bug was reported internally with the engineering team.

[hobby65]

Any update from our RedHat friends?

[bogdanmuresan]

Any update from our RedHat friends?

I have reopened the case to ask for updates (I had closed it initially when they said they filed an internal jira to fix the issue).
FIY, the issue is caused by "update_secrets: false" field at the user level, so you can remove those as a workaround.

the file "plugins/module_utils/aap_module.py" in ansible.platform collection is a copy of "plugins/module_utils/controller_api.py" from awx.awx file (https://github.com/ansible/awx/blob/a74e7301cd224555da3f90c921b82d3964c6731c/awx_collection/plugins/module_utils/controller_api.py#L896) but it's missing the fields_could_be_same method.

[hobby65]

Thnx for the feedback, removing the "update_secrets: false" as a workaround is ok in development.
Won't be taking this to production this way :-D
Will test further and report any new findings,

Thanks for the good work.

[djdanielsson]

This is a module issue and there isn't anything we can do in this collection, I guess I should have made that clear when I put the labels as blocked by module issue