QUIC gateway rapidly exceeds validator rate limits and hits disallowed

[sobl403]

When running Jet against a validator, I keep seeing outbound QUIC connections get accepted and then immediately closed by the validator with the disallowed application-close reason. Jet logs show rapid bursts of these disallowed disconnects, even though only a single Jet instance is running. This makes it impossible to keep a stable session with the validator and effectively blocks transaction forwarding. Please take a look why Jet keeps triggering disallowed closes.

Thank you!

[leafaar]

Thanks for the issue, I'm taking a look on this.

[leafaar]

Are you using a staked identity?

[leafaar]

Could you also provide some logs, that would help me understand better the behavior, thanks.

[sobl403]

Are you using a staked identity?

Yep

Could you also provide some logs, that would help me understand better the behavior, thanks.

Giving up sending transaction: *** to remote peer: ***, client identity: **, after 2 attempts: ConnectionError(ApplicationClosed(ApplicationClose { error_code: 2, reason: b"disallowed" }))

Giving up sending transaction: *** to remote peer: ***, client identity: **, after 2 attempts: ConnectionError(ApplicationClosed(ApplicationClose { error_code: 4, reason: b"too_many" }))

Also, when my validator is leader I see:
DEBUG solana_streamer::nonblocking::quic] Reject connection from <Jet SocketAddr> -- rate limiting exceeded

Upstream validators should recognize me as a staked connection, in my own validator logs I see:
solana_streamer::nonblocking::quic] Peer type Staked

I understand that some validators may have their own limit settings, but the problem occurs too often.

[leafaar]

How many txns do you send/sec? Also, how many connections are you using?

[sobl403]

How many txns do you send/sec? Also, how many connections are you using?

up to 20 transactions per second.

I don't know how many connections I'm using, that's what I'd like to know. Judging by the code, it's one connection per peer.

[leafaar]

Yes, but you can do up to 8 connection per peer

[leafaar]

Also, is it for all peers that you're having the disallowed error?

[sobl403]

Also, is it for all peers that you're having the disallowed error?

No, but about 500 validators.

The default validator settings allow 8 connections from a single IP address and 8 connections from a single identity. When the connection limit from a single IP address is exceeded, I should get a disallowed error, which is what I get most of the time. Much less often, I get a too_many error. I don't understand how and where I'm exceeding the connection limit from a single IP address.

[lvboudre]

@sobl403

disallow is triggered by the remote peer rate limiting that is based on IP rate limiting. (one IP cannot open more than 8 connections within a window).

Recent version of agave have increased the burst so we can see if you still have this issue in the near future (I think it will support 16 with higher burst).

That being said :

1. are you running jet OR multiple jet instance on the same host?
2. Are you running jet in the same host as your validator : both open many connections with peer, this higher chance of getting ratelimited.

[sobl403]

@lvboudre
are you running jet OR multiple jet instance on the same host? - No
Are you running jet in the same host as your validator : both open many connections with peer, this higher chance of getting ratelimited. - No

Do you see the problem now? Both the too_many and disallowed errors have become much more frequent in the new version. Now the logs show that workers for remote peer validators are dropping with errors. I'm trying to figure out what the problem is and would like to rule this out. Only the validator itself and two Jet instances are working with my validator key. I don't think the problem is that the validator is opening too many connections.

Also, for context, I'll say that for one validator, I first see too_many errors and reconnection attempts on one instance. After several unsuccessful connection attempts, I see the disallowed error.

[sobl403]

quic:
max_concurrent_connection: 1024
endpoint_count: 4

connection_prediction_lookahead: 10

There might be a problem with the config. Could you tell me if these values ​​are optimal for multiple instances?

[lvboudre]

@sobl403 after thorough testing the issue is in Agave and we've reached out Anza about this.
Sadly they probably gonna improve this in Q1 2026.

As for your configuration, for optimal performance its better to use 1:1 mapping between endpoint_count and max_concurrent_connection, so:

max_concurrent_connection: 1024
endpoint_count: 1024
connection_prediction_lookahead: 10

TLDR of the current connection problem:

Remote peer are dropping our connection during connection eviction process. Eviction happens when a validator does not have enough space in its internal connection table. If you are using staked jet instance, the eviction process is weighted-random (the more stake you have, less likely you are from being evicted), as for unstaked jet, latest connection get evicted.

Connection table is only 1000 capacity by default and people spam validators with new connection almost on each transaction they sent. We get disconnected, then we re-connect into order to drain all our transaction sender queue, then we get dropped again, repeat this 8 times and we get disallow. The eviction logic in agave right now is not smart enough to prune/evict spammer first.

Connection throttling is at IP address level so we cannot do anything about it unless you had multiple IP NIC to use.

I asked Anza to not penalize people who only open long-lasting connection.

[sobl403]

Thank you very much for your efforts and responsiveness!

[leafaar]

Thank you for opening this issue and providing valuable informations, please feel free to open as much issues you want.