Any plans for message priority levels and/or a message type governor?

[Eric24]

• Message Priority Levels: A numeric value from 0 to n, where higher numbers have higher priority. When reading messages from the queue, messages with higher priority values are returned before those with lower priority values. This allows for messages to be assigned relative priorities, similar to RabbitMQ Priority Queues.
• Message Type Governor: An optional property on the message that defines its "type" (arbitrary from the perspective of PGMQ). When reading (or as part of the queue configuration), the maximum number of messages with the same type value that can be "invisible" at one time is specified. So that if the max is 10, and there are already 9 invisible messages with type = 'xyz', only one more of this type can be returned, even if there are more and the reader asks for more. This allows for resource management so that one type of message can't use all available resources (i.e. if there are 20 available "message handlers", setting the max to 10 would prevent any given message type from using more than 50% of the handlers at once).

[ChuckHend]

Hi @Eric24,

There is not a formal plan in place for priority levels, although it's something that has been discussed and something I think would be worth supporting. We could work together on getting a plan in place for that! My first thought is that this might be best introduced as a new queue type, so the project would then have standard, partitioned, and now a priority queue. I'm open to ideas though.

A governor is an interesting idea. I have not personally seen something like this implemented but I can see where it would be useful, and feels kind of related to priority queue, but not exactly?

[knpwrs]

An alternative approach to priority is to have one queue per priority and then have one function that can return a message given a list of queues. For instance, consider this sample code from https://github.com/contribsys/faktory_worker_go:

// pull jobs from these queues, in this order of precedence
mgr.ProcessStrictPriorityQueues("critical", "default", "bulk")

// alternatively you can use weights to avoid starvation
mgr.ProcessWeightedPriorityQueues(map[string]int{"critical":3, "default":2, "bulk":1})

One big advantage of this approach is that you can avoid queue starvation (as noted in the second comment); i.e., lower-priority queues always get a chance to run no matter how many high-priority messages come in.

Another big advantage I think for pgmq in particular is that there wouldn't need to be a new queue type. Right now there is pgmq.read, I could conceivably imagine a parallel pgmq.read_weighted function that behaves like that second example from Faktory.

A disadvantage to this approach is that priorities must be defined ahead of time, so you can't have arbitrary priorities.