Separate request timeout for request and pool

[dimaqq]

Sometimes, the connection to APN just dies with much data in the pipe, in which case the last sent request is way ahead of last processed response:

Connection(host='api.push.apple.com', port=443, ᠁ channels={38907: Channel(wakeup=<asyncio.locks.Event object at 0x7f135abdb7f0 [unset, waiters:1]>,᠁}, ᠁closed=False, outcome=None, max_concurrent_streams=1000, last_stream_id_got=38907, last_stream_id_sent=38907

vs.

Channel(wakeup=<asyncio.locks.Event ᠁ [set]>, events=[<ResponseReceived stream_id:38645, headers:[(':status', '200'), ᠁]>, ]

Thus, 131 requests were lost in the void (Δ/2 as outbound requests have odd id's).

[dimaqq]

Proposed solution:

• posting request on a connection could have small timeout, e.g. 5s.
• posting request to a pool could have larger timeout, e.g. 30s.

So that, if one connection is stuck, the request can be retried on another connection.

If we go this way, however, we ought to assure that we won't accidentally try to send the request on the very same connection again. A couple of ideas off the cuff:

1. keep a liveliness statistic (sliding window avg response time, or instant requests waiting count), and select best/liveliest available connection for retry, or
2. keep a weakset of used connections in the request retry loop and reject connections that were already tried

Option 1. is pre-emptive, and will balance out connections as they get stuck and unstuck.
Option 2. is more straight-forward and robust.

🤔