Rework the pipeline: batch allocation + deterministic backoff

[wyfo]

See more details in #1769 (comment)

[wyfo]

The performance regression on Linux has been fixed.
The random backoff has been replaced by a deterministic batching mode: when there is more than one batches used at the same time (for a given priority), it means that the tx task is not delivering messages fast enough, so the pipeline enters in batching mode where it doesn't notify the tx task when messages are written on the current batch.
On the tx side, when batching mode is detected, it backs off until a deadline starting from the latest successful pull (or until a batch is pushed ofc); if deadline is reached, batching mode is manually stopped (next notification will be triggered on the pipeline side). If there is no backoff (or deadline was reached), the current buffer is retrieved — or a small backoff is applied if it is not available, but a notification should arrive soon in this case.

This PR gives the same throughput as main branch on Linux, around 2.7Mmsg/s on my Linux setup. On my Mac, both branches are around 6Mmsg/s. Latency of low throughput is untouched. The CPU consumed by the tx task seems to have been improved, which was expected because batching is done more deterministically and systematically.
Regarding memory consumption, batches are allocated with a size depending on the current workflow: if the last pushed batch was small enough, then the small size (an arbitrary constant of 2kiB for now) is used for the next one, otherwise, it will use the max size.

The current PR version embeds a batch reusing mechanism: one single batch is kept in memory, and if it is not available (meaning it is currently owned by the tx task), a new one is allocated on demand, respecting the limit fixed by the configuration. The batch will be refilled by the tx task after use, the next one being dropped if the slot is already refilled.
However, the impact of the allocations on performance is quite negligible, as reported on the graph below: when refilling is commented out, malloc in get_batch counts for only 0.01% of the CPU, and free in refill for 0.1%. As expected with these numbers, not using refilling doesn't have impact on performance.

Here are the flamegraphs of z_pub_thr

• this PR
  
• this PR (without using batch refill, so only allocations)
  
• main
  

[wyfo]

This constant is of course arbitrary. It can be exposed in configuration if needed.

[wyfo]

I've reverted the previous commit, so the numbers are back to the original ones. Throughputs of big messages was indeed very impacted by a smaller batch number, 130kmsg/s -> 50kmsg/s for 70kB messages.
@Mallets Do you think we should modify the numbers? for example increasing the hight priority to 2? In fact this PR does lazy allocation, and only keep one buffer in memory, so we could put any numbers, it would not change memory consumption at all, it just a matter of throughput/congestion.

[OlivierHecart]

The numbers are not coherent with commons/zenoh-config/src/defaults.rs.

[wyfo]

You're right, good catch !

[codecov]

Codecov Report

Attention: Patch coverage is 95.80713% with 20 lines in your changes missing coverage. Please review.

Please upload report for BASE (main@1964079). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
io/zenoh-transport/src/common/pipeline.rs	95.70%	20 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1769   +/-   ##
=======================================
  Coverage        ?   70.45%           
=======================================
  Files           ?      359           
  Lines           ?    64735           
  Branches        ?        0           
=======================================
  Hits            ?    45608           
  Misses          ?    19127           
  Partials        ?        0           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.