Schedule job in created and active state

[roelandwyns]

Hi timgit,

First of all, thank you for this amazing job scheduling tool!

We have a new use case coming up for which we would like to use pg-boss.
We would like to schedule jobs with a kind of singleton key, but allow 1 job to be added in the created state when there is already one active. (And discard all the extra ones). At the moment the singleton key prevents an extra job to be in the created state due to a DB constraint. If a job is active, no new one will be scheduled.

A bit more into why we would need this: some data gets modified on our platform, and as a result a lot of features need to be recalculated.
This recalculation should be done in a background job, as it is too heavy. However, while this job is running with the recalculation, it is still possible that the user modifies the data on out platform again, and a new job would be needed to do the recalculation. That's why we would like to schedule the job in the created state as well.
I looked into the singleton key option, but that does not allow scheduling a next job when another one is already active. (In the job itself, we can poll for the latest data, so one in active and one in created is enough. No new jobs need to be scheduled when there is already one in created state.)

Do you have any ideas/suggestions on how to tackle this with pg-boss? If not possible with pg-boss, we have a back-up solution without it, but it would be a lot cleaner with pg-boss.

Thank you already,

Roeland

[timgit]

There are a couple of options in the API that you can look into: publishDebounced() and publishSingleton(). Let me know if you think either of these would help out in your use case

[roelandwyns]

Hi Timgit,

Thank you for the quick reply!
I think publishSingleton won't fit our use case, as it does not allow a job to be added with the same singleton key in the created state when there is already a job running?
I thought so when I read the documentation, but when diving into the code it did the same as publishOnce it seems. Also tested it and it doesn't make a new job with the same singletonkey when there is already one in active.

Our usage depends on an action from a user to decide if we need a new job or not, not really based on time.

The current idea we are running with is to use the onComplete callback from the job, to verify if everything is up to date and if not, schedule a new job from that point.

You can close this discussion if you want as that solution should be enough for our use case. However it would be cool to have the option to have an active job and a same job in the created state possible.

[timgit]

publishSingleton() uses a different constraint with a hard-coded key to skip over active jobs. This was added specifically to prevent jobs from being queued if the queue has at least 1 job in the queue. There's a variety of ways to build what you're wanting with the queues, including onComplete like you mentioned. Hope this helped. :)

[roelandwyns]

Hi Timgit,

One last question: do you think it would be possible to support multiple queues in publishSingleton as well? (Now it is with the hard-coded key to skip over active jobs indeed), but would be cool to have multiple queues

[timgit]

Yes, this would work with multiple queues if another unique constraint is introduced to support that specifically. Until then, please investigate using wildcards in your queue names along with publishSingleton() and see if this is a good intermediate solution.

[adamhamlin]

+1 for multiple queues. I think there are lots of jobs where a publisher is ok skipping it if another one is already queued up, but not ok skipping if that other one is already in progress. Can lead to a lot of unnecessary processing. But I would also like to avoid having an explosion of channels that all basically do the same thing (UpdateResource_1, UpdateResource_2, etc). That information should probably only be encoded in the job data, not the name.

I wonder if you could just update the existing indexes to be where singletonKey startsWith SINGLETON_QUEUE_KEY instead of where equal to.

[timgit]

Intriguing suggestion with the pattern matching in the unique constraint. 👍

[adamhamlin]

@timgit would you be interested in a PR for this?

[timgit]

Yes! Thanks

[adamhamlin]

@timgit just in case you didn't get notified, I did open #329

[timgit]

Got it!

[timgit]

@adamhamlin just released this. Thanks!

[adamhamlin]

@timgit awesome! You're welcome, thank you!