Unable to upgrade AWX

[dangoncalves]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to security@ansible.com instead.)

Bug Summary

I try to upgrade AWX from 21.13.0 to 23.7.0. I upgrade the operator and the AWX versions at the same time. Then my upgrade is stuck at migration process. If I try to run migrations manually from a task pod I get the following error:

bash-5.1$ awx-manage migrate                                                                                                                                                                                                                   
Operations to perform:                         
  Apply all migrations: auth, conf, contenttypes, main, oauth2_provider, sessions, sites, social_django, sso  
Running migrations:                        
  Applying conf.0010_change_to_JSONField...Traceback (most recent call last):                        
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)                                                                                    
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)                                                                                      
psycopg.errors.InvalidTextRepresentation: invalid input syntax for type json
DETAIL:  Token "https" is invalid.                                                                                     
CONTEXT:  JSON data, line 1: https...
                                                           
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/awx-manage", line 8, in <module>
    sys.exit(manage())
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py", line 175, in manage
    execute_from_command_line(sys.argv)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv) 
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 412, in run_from_argv 
    self.execute(*args, **cmd_options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/base.py", line 106, in wrapper
    res = handle_func(*args, **kwargs)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/core/management/commands/migrate.py", line 356, in handle
    post_migrate_state = executor.migrate(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 135, in migrate
    state = self._migrate_all_forwards(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 167, in _migrate_all_forwards
    state = self.apply_migration(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/executor.py", line 252, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/migration.py", line 132, in apply
    operation.database_forwards(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/migrations/operations/fields.py", line 235, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py", line 830, in alter_field
    self._alter_field(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/schema.py", line 287, in _alter_field
    super()._alter_field(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py", line 1055, in _alter_field
    self.execute(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/postgresql/schema.py", line 48, in execute
    return super().execute(sql, None)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/base/schema.py", line 201, in execute
    cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/db/backends/utils.py", line 87, in _execute
    return self.cursor.execute(sql)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/psycopg/cursor.py", line 723, in execute
    raise ex.with_traceback(None)
django.db.utils.DataError: invalid input syntax for type json
DETAIL:  Token "https" is invalid.
CONTEXT:  JSON data, line 1: https...

AWX version

21.13.0

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

1. Install 21.13.0
2. Upgrade operator and AWX to last version
3. Connect to a task pod kubectl exec -it <task_pod> -- bash
4. Run the migration command awx-manage migrate

Expected results

Migrations executes without any error

Actual results

Migration conf.0010_change_to_JSONField raise an exception

Additional information

No response

[fosterseth]

this likely means you have some data in your Settings that is not properly JSON (but may have been allowed on version 21.13.0)

the https: indicates it is one of your URL related setting. One workaround might be to clear out your Settings in UI (the ones that have https in the values) and then restore them (after making them proper JSON)?

[fosterseth]

also if you can snapshot and redact your settings, we might be able to point out which one is the culprit

[dangoncalves]

Hello @fosterseth,

Thank you for your help and your insights.

I cleared all my settings in ui and I cannot modify these:

awx setting list | grep -i https
     "TOWER_URL_BASE": "https://my.domain.name.com",
     "AUTOMATION_ANALYTICS_URL": "https://example.com",
     "SOCIAL_AUTH_GOOGLE_OAUTH2_CALLBACK_URL": "https://my.domain.name.com/sso/complete/google-oauth2/",
     "SOCIAL_AUTH_GITHUB_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github/",
     "SOCIAL_AUTH_GITHUB_ORG_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github-org/",
     "SOCIAL_AUTH_GITHUB_TEAM_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github-team/",
     "SOCIAL_AUTH_GITHUB_ENTERPRISE_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github-enterprise/",
     "SOCIAL_AUTH_GITHUB_ENTERPRISE_ORG_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github-enterprise-org/",
     "SOCIAL_AUTH_GITHUB_ENTERPRISE_TEAM_CALLBACK_URL": "https://my.domain.name.com/sso/complete/github-enterprise-team/",
     "SOCIAL_AUTH_AZUREAD_OAUTH2_CALLBACK_URL": "https://my.domain.name.com/sso/complete/azuread-oauth2/",
     "SOCIAL_AUTH_SAML_CALLBACK_URL": "https://my.domain.name.com/sso/complete/saml/",
     "SOCIAL_AUTH_SAML_METADATA_URL": "https://my.domain.name.com/sso/metadata/saml/",

Unfortunately, it does not solve the issue.

Moreover, upgrading to 22.3.0 works while 22.4.0 breaks the upgrade with the migration error.

[fosterseth]

yeah 22.4.0 is where we switched over to django's built-in JSONField

nothing in those values look strange. In fact those are just the default values. I couldn't replicate the issue on my hand sadly

[dangoncalves]

Hello @fosterseth

Unfortunately I wasn't able to reproduce on my side too. Something should be wrong in the settings but I can't figure out what. Any insight to help me finding the culprit setting?

[dangoncalves]

Finally I was able to figure out what was going wrong. My TOWER_URL_BASE setting were corrupted in database: it were registered without quotes.

Many thanks for your help @fosterseth