feat: add per-project rate limiter settings override #583

zmoog · 2025-05-29T06:30:00Z

Add a per-project override mechanism for rate limiter settings.

Changes:

Add a new ratelimit.overrides option
Add a new ratelimit.throttle_interval option

For example, in the following config:

ratelimit/overrides_throttle_interval:
  rate: 100
  burst: 200
  strategy: bytes
  throttle_behavior: error 
  throttle_interval: 10s # ✨New option
  overrides: # ✨New option
    x-elastic-project-id:e678ebd7-3a15-43dd-a95c-1cf0639a6292:
      rate: 400
      burst: 800
      throttle_interval: 42s

In overrides, you can override one or more of the following settings:

rate
burst
throttle_interval

If not overridden, the base settings apply.

Examples

(1) Simple rate limiter settings (no override)

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error

Notes on rate and strategy:

Strategy	`rate` value
`requests`	Requests per second
`records`	Records per second
`bytes`	Bytes per seconds

(2) Per-project rate override

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline # 👀

(3) Per-project rate and burst override

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline # 👀
    x-elastic-project-id:whatever:
      rate: 3070230 # +50% over baseline # 👀
      burst: 7675575 # +50% over baseline # 👀

(4) Set base throttle_interval

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  throttle_interval: 10s # 👀
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline

(5) Set base throttle_interval to 10s and override it on a project

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  throttle_interval: 10s # 👀
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline
    x-elastic-project-id:whatever:
      throttle_interval: 15s # 👀

zmoog · 2025-06-02T06:15:09Z

@simitt, I'd like to double-check with you whether this is the override semantics we're looking for.

Note: I also discussed the switch from a request-based to a bytes-based ¹ rate limiting with @vigneshshanmugam. So, in the following examples, I will use the bytes strategy to reflect this.

Let's start with a few examples.

(1) Baseline rate-limiting

In this example, the rate limiter applies the same baseline limits to all projects.

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error

(2) Per-project rate override

In this example, we increase the rate by 50% over the baseline.

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline

(3) Per-project rate and burst override

In this example, we increase the rate and burst by 50% over the baseline.

ratelimit:
  metadata_keys:
   - x-elastic-project-id
  rate: 2046820
  burst: 5117050 # In these examples, I'm considering a 2.5x burst ratio 
  strategy: bytes
  throttle_behavior: error
  overrides:  
    x-elastic-project-id:local:
      rate: 3070230 # +50% over baseline
    x-elastic-project-id:whatever:
      rate: 3070230 # +50% over baseline
      burst: 7675575 # +50% over baseline

I noticed the comment in the bytes strategy that highlights that the using this strategy is "much more expensive" than that the requests and records strategies. We should consider this aspect, and run some tests to measure this increase — if we don't already have some. ↩

simitt · 2025-06-02T07:06:05Z

General approach looks good to me. What would be worth adding is a time interval indicator, e.g. rate_per_second if the rate is applied on a per second interval.
Gubernator also allows to configure the throttling interval. I don't think this is currently set anywhere, and is not the most urgent, but would be a nice add-on to also allow for it. See an example for this in https://github.com/elastic/serverless-gitops/blob/main/services/apm-ingest-service/values/production/default.yaml.

zmoog · 2025-06-04T11:10:26Z

General approach looks good to me. What would be worth adding is a time interval indicator, e.g. rate_per_second if the rate is applied on a per second interval. Gubernator also allows to configure the throttling interval. I don't think this is currently set anywhere, and is not the most urgent, but would be a nice add-on to also allow for it. See an example for this in https://github.com/elastic/serverless-gitops/blob/main/services/apm-ingest-service/values/production/default.yaml.

Adding a throttle_interval seems a low hanging fruit. Adding it!

constanca-m

Thank you @zmoog ! It looks great, I couldn't find issues with your approach.

constanca-m · 2025-06-05T07:54:43Z

processor/ratelimitprocessor/config.go

+// If no override is found, the default rate limit settings are returned.
+func resolveRateLimitSettings(cfg *Config, uniqueKey string) RateLimitSettings {
+	result := cfg.RateLimitSettings
+	if len(cfg.Overrides) > 0 {


This condition wouldn't be necessary, right? Just doing if override, ok := cfg.Overrides[uniqueKey]; ok { would be enough.

Not needed. This optimization isn't worth sacrificing readability for. I'm removing it.

constanca-m · 2025-06-05T07:57:14Z

processor/ratelimitprocessor/config.go

+	var errs []error
+	if r.Rate != nil {
+		if *r.Rate <= 0 {
+			errs = append(errs, fmt.Errorf("rate must be greater than zero"))


You know... if we were pushing this upstream, and we ran make lint, it would complain that we should use errors.New instead of fmt.Errorf in this case since we are not using any variables in the message :) You don't have to change it, just something that I noticed.

Oh, thank you Constança! I forgot to run make lint before committing the changes 🤦

vigneshshanmugam

LGTM overall

processor/ratelimitprocessor/config.go

Users can now set a default value for `throttle_interval` and also override it per project when needed. It only has an effect when the rate limiter type is `gubernator`.

It can save us the key hashing to access the map, but I'm not sure it's worth it.

Co-authored-by: Vignesh Shanmugam <[email protected]>

vigneshshanmugam

LGTM

vigneshshanmugam · 2025-06-06T17:22:46Z

processor/ratelimitprocessor/README.md

+  ratelimiter:
+    metadata_keys:
+    - x-elastic-project-id


To keep it generic, lets use x-tenant-id or similar.

vigneshshanmugam · 2025-06-06T17:23:41Z

processor/ratelimitprocessor/README.md

 | `type`              | Type of rate limiter. Options are `local` or `gubernator`.                                                                                                                                                        | No       | `local`    |
+| `overrides`         | Map of metadata key overrides for the rate limiter. See the possible overrides and examples below.                                                                                                                | No       |            |
+
+### Possible overrides:


nit: Overrides based on Metadata keys?

zmoog force-pushed the zmoog/feat/allow-per-project-overrides branch from 115e243 to 6395814 Compare June 4, 2025 11:08

zmoog changed the title ~~Add ratelimiter per-project settings override~~ rate limiter: add per-project settings override Jun 4, 2025

constanca-m reviewed Jun 5, 2025

View reviewed changes

zmoog force-pushed the zmoog/feat/allow-per-project-overrides branch from 98e1d0b to 520fe46 Compare June 5, 2025 09:43

zmoog changed the title ~~rate limiter: add per-project settings override~~ feat: add per-project rate limiter settings override Jun 5, 2025

zmoog marked this pull request as ready for review June 5, 2025 13:54

zmoog requested a review from a team as a code owner June 5, 2025 13:54

zmoog requested review from jackshirazi and edmocosta June 5, 2025 13:54

vigneshshanmugam reviewed Jun 6, 2025

View reviewed changes

processor/ratelimitprocessor/config.go Show resolved Hide resolved

processor/ratelimitprocessor/config.go Outdated Show resolved Hide resolved

processor/ratelimitprocessor/config.go Show resolved Hide resolved

zmoog and others added 12 commits June 6, 2025 13:12

Draft ratelimiter per project settings override

b8dc014

Fix errors left after the rebase

75fb9ec

Use RateLimitSettings as a template to override

c4faa9f

Add more tests

6911fdc

Update tests

2bbd93a

Add throttle_interval config option

69e26ea

Users can now set a default value for `throttle_interval` and also override it per project when needed. It only has an effect when the rate limiter type is `gubernator`.

Remove optimization

019790a

It can save us the key hashing to access the map, but I'm not sure it's worth it.

We use errors.New() when there are no variables

bc918ef

Update processor/ratelimitprocessor/config.go

88348e1

Co-authored-by: Vignesh Shanmugam <[email protected]>

Update RateLimitOverrides struct name

53c778f

Use errors.New() when there are no variables

7621cf2

Fix missed renames

dc9e4d4

zmoog force-pushed the zmoog/feat/allow-per-project-overrides branch from 03955c2 to dc9e4d4 Compare June 6, 2025 11:12

zmoog added 4 commits June 6, 2025 13:38

Update README with override options

71ec56a

Fix & clarify README

1dc9fd6

Minor cleanup

00dc78b

Add an example for the throttle_interval option

7a9f693

vigneshshanmugam approved these changes Jun 6, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add per-project rate limiter settings override #583

feat: add per-project rate limiter settings override #583

zmoog commented May 29, 2025 •

edited

Loading

Uh oh!

zmoog commented Jun 2, 2025

Uh oh!

simitt commented Jun 2, 2025

Uh oh!

zmoog commented Jun 4, 2025

Uh oh!

constanca-m left a comment

Uh oh!

constanca-m Jun 5, 2025

Uh oh!

zmoog Jun 5, 2025

Uh oh!

constanca-m Jun 5, 2025

Uh oh!

zmoog Jun 5, 2025

Uh oh!

vigneshshanmugam left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vigneshshanmugam left a comment

Uh oh!

vigneshshanmugam Jun 6, 2025

Uh oh!

vigneshshanmugam Jun 6, 2025

Uh oh!

Uh oh!

feat: add per-project rate limiter settings override #583

Are you sure you want to change the base?

feat: add per-project rate limiter settings override #583

Conversation

zmoog commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Examples

(1) Simple rate limiter settings (no override)

(2) Per-project rate override

(3) Per-project rate and burst override

Uh oh!

zmoog commented Jun 2, 2025

Footnotes

Uh oh!

simitt commented Jun 2, 2025

Uh oh!

zmoog commented Jun 4, 2025

Uh oh!

constanca-m left a comment

Choose a reason for hiding this comment

Uh oh!

constanca-m Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

zmoog Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

constanca-m Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

zmoog Jun 5, 2025

Choose a reason for hiding this comment

Uh oh!

vigneshshanmugam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vigneshshanmugam left a comment

Choose a reason for hiding this comment

Uh oh!

vigneshshanmugam Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

vigneshshanmugam Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zmoog commented May 29, 2025 •

edited

Loading