AAP-39559 Wait for all event processing to finish, add fallback task #15798
Conversation
|
| right_now_time = now() | ||
| window_end = right_now_time - timedelta(seconds=settings.INDIRECT_HOST_QUERY_FALLBACK_MINUTES) | ||
| window_start = right_now_time - timedelta(days=settings.INDIRECT_HOST_QUERY_FALLBACK_GIVEUP_DAYS) | ||
| for job in Job.objects.filter(event_queries_processed=False, finished__lte=window_end, finished__gte=window_start).iterator(): |
There was a problem hiding this comment.
When this runs the first time, it could lead to a large number of processing jobs (all jobs completed in the last three days). Are we ok with that? Most of them will probably end quickly because there are no queries.
There was a problem hiding this comment.
Well that seems like a good argument for having a limit to the start of the window which I did here. Otherwise that was more of a gut feeling.
But I still sense something to be worried about. This will get scheduled for a lot of jobs that don't need the processing done. The periodic task will run and schedule a job for each unprocessed job. Each should be a no-op with a single write, but that could still be a lot of jobs. Worse, it could temporarily overwhelm the dispatcher, because it can't delay tasks (ideal might be to scramble them with random delays).
I have a thought for this, and I'll work on putting something up. With a different angle, it'll get more intuitive.
|
@AlanCoding going to merge this, so I can add my code |
pb82
merged commit bc08e03
into
ansible:feature_indirect-host-counting
…15798) * Wait for all event processing to finish, add fallback task * Add flag check to periodic task
SUMMARY
Flaw of logic up until this point was that we could be missing events. This does the stuff needed to close those loopholes by adding a task to pick up processing of yet-unprocessed jobs. This does some windowing just so that the task doesn't accidentally blow up in unforeseen circumstances.
@pb82 You should rename the periodic task added in this, and then piggyback the cleanup code on top of this. That way there's only 1 periodic task running for the management of these things.
ISSUE TYPE
COMPONENT NAME