#2172 - Retry Push for certain HTTP status codes

[k-macmillan]

Description

issue #2172

The goal of this PR is to move logic to send VEText push notifications into a celery tasks. We added retry logic for the following statuses: 429, 500, 502, 503, 504.

#Changes in This PR

V2 Push Notifications

• Update status code for service unauthorized from 401 to 403
• Updated push notification flow:
  • Validates requests before queuing deliver_push and returns V2PushPayload dataclass.
  • Formatted request for VEText using format_for_vetext and returns dict.
  • Validated payload dict sent to deliver_push celery task

Celery Provider Tasks

• Added deliver_push celery task to handle push notifications using vetext_client, instead of being handled directly by the API.
• Routed deliver_push to SEND_EMAIL queue.
• Implemented retry logic with AutoRetryException for transient failures.

VEText Client

• Created format_for_vetext for formatting VEText Payload
• Update logic for retryable exceptions - 429, 500, 502, 503, 504 status codes, ConnectionTimeout, ReadTimeout

Config Updates

• Registered app.celery.provider_tasks in Celery imports.
• Add VETEXT_USERNAME and VETEXT_PASSWORD to celery task definitions

Internal REST API

• Added /502 and /sleep endpoints for debugging.

Mobile App Registry

• Improved error handling: Logs & raises when accessing unregistered apps.

How Has This Been Tested?

• Deploy OK

Test Happy Path Push 201

• Send POST request to /v2/notifications/push
• Confirm logs VEText response 200
• Confirm 201 from /v2/notification/push
  

Test Happy Path Push Broadcast 201

• Send POST request to /v2/notifications/push/broadcast
• Confirm logs VEText response 200
• Confirm 201 from /v2/notification/push
  

Test Sad Path VEText Service Not Authorized 403

• Remove push permissions from service
  

• Confirm 403 from /v2/notification/push when service unauthorized

  • Confirm in logs 403
  

Test Sad Path VETex Retrible Failure

• Remove celery task definition for VETEXT_PASSWORD and deploy changes
• Send normal request to /v2/notifications/push
• Checked logs to confirm 500 response from VETest and push notification retried.
• Add back VETEXT_PASSWORD and redeploy
• Check logs to confirm push notification now sent after redeploy

Checklist

• I have assigned myself to this PR
• PR has an appropriate title: #9999 - What the thing does
• PR has a detailed description, including links to specific documentation
• I have added the appropriate labels to the PR.
• I did not remove any parts of the template, such as checkboxes even if they are not used
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to any documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works. Testing guidelines
• I have ensured the latest main is merged into my branch and all checks are green prior to review
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• The ticket was moved into the DEV test column when I began testing this change

[cris-oddball]

If you redact the ICN here, won't the retry attempt to use <redacted> instead of the icn? Or does this just affect the logs?

[kalbfled]

That's a good question.

[MackHalliday]

Updated to deepcopy the payload before redact

Retested

https://oddball.slack.com/archives/C06K27HSRL6/p1739222921565339?thread_ts=1738869638.085009&cid=C06K27HSRL6

[kalbfled]

That's a good question.

[kalbfled]

The debug log duplicates output from the preceding info log. Do we need to log the response json or status code twice in succession? Would including only the payload in the debug output be sufficient?

[MackHalliday]

Yes - This make is so much easy for the sake of testing - easy to link . We need the info log for upper environments .

[kalbfled]

Ditto Cris's question about changing the ICN value to "redacted". That's done twice in this file.

[MackHalliday]

Updated.