Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Worker failure appears to increment retry counter #1015

Open
kmcgovern-apixio opened this issue Jan 29, 2025 · 0 comments
Open

[BUG] Worker failure appears to increment retry counter #1015

kmcgovern-apixio opened this issue Jan 29, 2025 · 0 comments
Assignees
Labels
bug Something isn't working

Comments

@kmcgovern-apixio
Copy link

kmcgovern-apixio commented Jan 29, 2025

Describe the bug
When a worker fails a task will move into archived state without retrying if asynq.MaxRetries(0) is set on the task. Setting it to any positive value causes allows X worker failures. IE setting asynq.MaxRetries(3) will cause allow the task to get picked up again up to 3 times with worker failures as the cause

Environment (please complete the following information):

  • OS: linux
  • asynq package version: 0.25.1
  • Redis version: 7.4.2

To Reproduce
Steps to reproduce the behavior (Code snippets if applicable):

  1. create task with asynq.MaxRetries(0)
  2. run task
  3. kill worker before tasks completes (IE ctrl + c)
  4. check task state in redis hgetall "asynq:{default}:t:8abc3be7-2f6f-4ac4-a610-1c0a23188d96"
  5. start worker backup
  6. after lease expires, task will move to archived state (can check with hgetall)

Expected behavior
tasks are picked back up on worker failure. retries is incremented only upon a task returning an error or panic

Screenshots
output from redis cli (payloads and job name redacted)

127.0.0.1:6379> hgetall "asynq:{default}:t:8abc3be7-2f6f-4ac4-a610-1c0a23188d96"
1) "state"
2) "active"
3) "msg"
4) "\n\x1finternal:myjobname\x12\xc2\x02taskpayloadhere\x1a$8abc3be7-2f6f-4ac4-a610-1c0a23188d96\"\adefault@\x88\x0e`\x80\xa3\x05"
127.0.0.1:6379> hgetall "asynq:{default}:t:8abc3be7-2f6f-4ac4-a610-1c0a23188d96"
1) "state"
2) "archived"
3) "msg"
4) "\n\x1finternal:myjobname\x12\xc2\x02taskpayloadhere\x1a$8abc3be7-2f6f-4ac4-a610-1c0a23188d96\"\adefault:\x19asynq: task lease expired@\x88\x0eX\xa6\xe7\xe6\xbc\x06`\x80\xa3\x05"

Additional context
For now i am bumping the max retry value to be able to handle this, but I was expecting worker failures to not impact retries

@kmcgovern-apixio kmcgovern-apixio added the bug Something isn't working label Jan 29, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants