HTTP2 multiple connection pooling

[michaelfeil]

Hey,

I am maintainer of a couple of OSS projects and recenty released a py03 package that allows for a "pyrthon fanout" to send requests to a microservice load-balancer.
https://github.com/basetenlabs/truss/tree/main/baseten-performance-client

When upgrading to http2, I noticed that there are two kind of performance degradations when switching from http1 to http2.
Under low concurrency (e.g. 1/2/4/8) the http2 reqwest client performs better.
Under high concurrency (64) the http1 client starts to catch up.
Under high number of requests (1024), with a semaphore of say 128, the client performs worse than e.g. 8x 128. For http1, the client performs exactly identical. I updated the http2_initial_stream_window_size + http2_initial_connection_window_size to 2MB each (payloads are ~2kb, so no bottleneck). I also set the http2_max_frame_size.

The result would be to have some kind of max-concurrency per open tcp connection, and allowing multiple streams, and a pool of connections. e.g. a suitable fix would be to have a pool of 32 clients and queueing a new request to the least use one.

[michaelfeil]

basetenlabs/truss#1769

[seanmonstar]

It's likely you're hitting some contention in the h2 library. It's a known issue, and recently has some effort into improving it (most recent comments): hyperium/h2#531

Trying to add support directly in reqwest (well, hyper-util's legacy pool) to use multiple http2 connections would be quite a bit of work, and I'd rather focus that work on the newer pool implementations.

If I was wanting to accomplish what you wanted right now, I'd probably construct the layers of reqwest that I needed manually (like from tower/tower-http), and put the tower::balance::p2c::Balance layer just above the hyper-util legacy client, and feed it however many nodes I want to balance over.