Skip to content

Conversation

@cratelyn
Copy link
Member

@cratelyn cratelyn commented Nov 6, 2024

feat(app): Route frame count metrics

⛅ overview

this introduces a new tower middleware for Prometheus metrics, used for
instrumenting HTTP and gRPC request bodies, and observing (a) the
number of frames yielded by a body, (b) the number of bytes included
in body frames, and (c) a distribution of the size of frames yielded.

this builds upon the backend-level metrics added in #3308. this
additionally uses the route label extractor, hoisted out of the retry
middleware's Prometheus telemetry in #3337.

📝 changes

  • a linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData
    middleware is added, which complements the equivalent
    linkerd_http_prom::body_data::response middleware.

  • this is added to policy routes' metrics layer.

see prometheus/client_rust#241 and prometheus/client_rust#242, which
track upstream proposals to add accessors to Histogram that will allow
us to make test assertions that metrics are working properly. for now,
these are feature gated as also done in #3308.


🔗 related

@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 378e494 to 843f57c Compare November 7, 2024 06:00
@cratelyn cratelyn changed the title feat(app): Route request frame count metrics feat(app): Route request frame count metrics (wip) Nov 7, 2024
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 843f57c to 82d97be Compare November 7, 2024 06:37
@cratelyn cratelyn changed the base branch from kate/record-body-data to kate/hoist-route-label-extractor November 7, 2024 06:37
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from cc0e8f0 to 67d56a0 Compare November 10, 2024 22:29
@cratelyn cratelyn force-pushed the kate/hoist-route-label-extractor branch from 2d8d0f2 to a75492c Compare November 10, 2024 22:37
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch 2 times, most recently from 3ae2592 to 082cb90 Compare November 10, 2024 22:39
@cratelyn cratelyn changed the title feat(app): Route request frame count metrics (wip) feat(app): Route frame count metrics Nov 10, 2024
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 082cb90 to 878881d Compare November 10, 2024 23:07
@cratelyn cratelyn marked this pull request as ready for review November 11, 2024 01:26
@cratelyn cratelyn requested a review from a team as a code owner November 11, 2024 01:26
@cratelyn cratelyn force-pushed the kate/hoist-route-label-extractor branch 2 times, most recently from a84a157 to 71cca7f Compare November 21, 2024 14:50
Base automatically changed from kate/hoist-route-label-extractor to main November 21, 2024 14:56
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 878881d to 8868a39 Compare November 21, 2024 15:01
@cratelyn cratelyn marked this pull request as draft November 21, 2024 15:02
@cratelyn
Copy link
Member Author

rebased onto main now that #3308 and #3337 have landed, going to apply the same transformation to this as performed in review of #3308 here, and then re-open this as ready for review.

@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 8868a39 to 389f7dc Compare November 21, 2024 16:09
 ### ⛅ overview

this introduces a new tower middleware for Prometheus metrics, used for
instrumenting HTTP and gRPC request bodies, and observing (a) the
number of frames yielded by a body, (b) the number of bytes included
in body frames, and (c) a distribution of the size of frames yielded.

this builds upon the backend-level metrics added in #3308. this
additionally uses the route label extractor, hoisted out of the retry
middleware's Prometheus telemetry in #3337.

 ### 📝 changes

* a `linkerd_http_prom::body_data::request::NewRecordBodyData::NewRecordBodyData`
  middleware is added, which complements the equivalent
  `linkerd_http_prom::body_data::response` middleware.

* this is added to policy routes' metrics layer.

see prometheus/client_rust#241 and prometheus/client_rust#242, which
track upstream proposals to add accessors to `Histogram` that will allow
us to make test assertions that metrics are working properly. for now,
these are feature gated as also done in #3308.

Signed-off-by: katelyn martin <[email protected]>
@cratelyn cratelyn force-pushed the kate/record-body-data-part-deux branch from 389f7dc to 94f39e4 Compare November 21, 2024 16:36
@cratelyn cratelyn marked this pull request as ready for review November 21, 2024 16:44
@cratelyn cratelyn requested a review from olix0r November 21, 2024 16:45
Copy link
Member

@olix0r olix0r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌮 🎉

@cratelyn cratelyn merged commit e4d7d57 into main Nov 21, 2024
15 checks passed
@cratelyn cratelyn deleted the kate/record-body-data-part-deux branch November 21, 2024 19:43
cratelyn added a commit that referenced this pull request Sep 16, 2025
#3334 introduced tower
middleware to the outbound proxy that records the size and number of
response body's data frames.

this commit introduces an equivalent layer to the inbound proxy's
prometheus metrics.
cratelyn added a commit that referenced this pull request Sep 16, 2025
#3334 introduced tower
middleware to the outbound proxy that records the size and number of
response body's data frames.

this commit introduces an equivalent layer to the inbound proxy's
prometheus metrics.

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit that referenced this pull request Nov 12, 2025
this is akin to #3334, which cleaned up the equivalent expression in
`linkerd/app/outbound/src/http/logical/policy/route/metrics.rs`.

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit to linkerd/website that referenced this pull request Nov 14, 2025
the documentation of our proxy metrics has not kept pace with all of the
exciting telemetry that has been introduced since
#1599 documented the state of our
authorization policy metrics.

this commit reworks the documentation to exhaustively document the
families of metrics exported by the proxy. this commit does not
introduce mention of the _inbound_ metrics that have been added, but
does rename this section to "_Endpoint Metrics_" in order to be
compatible with the future addition of `inbound_http_route_*`,
`inbound_http_route_backend_*`, `inbound_grpc_route*`, and
`inbound_grpc_route_backend_*` metrics.

see:
* linkerd/linkerd2-proxy#2377
* linkerd/linkerd2-proxy#2380
* linkerd/linkerd2-proxy#3086
* linkerd/linkerd2-proxy#3308
* linkerd/linkerd2-proxy#3334

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit that referenced this pull request Nov 14, 2025
this is akin to #3334, which cleaned up the equivalent expression in
`linkerd/app/outbound/src/http/logical/policy/route/metrics.rs`.

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit that referenced this pull request Nov 14, 2025
this is akin to #3334, which cleaned up the equivalent expression in
`linkerd/app/outbound/src/http/logical/policy/route/metrics.rs`.

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit that referenced this pull request Nov 14, 2025
this is akin to #3334, which cleaned up the equivalent expression in
`linkerd/app/outbound/src/http/logical/policy/route/metrics.rs`.

Signed-off-by: katelyn martin <[email protected]>
cratelyn added a commit that referenced this pull request Nov 17, 2025
this branch makes some small preliminary changes to the outbound proxy's
backend metrics layer.

this will simplify the introduction of additional middleware layers.

---

* refactor(app/outbound): simplify backend layer construction

this is akin to #3334, which cleaned up the equivalent expression in
`linkerd/app/outbound/src/http/logical/policy/route/metrics.rs`.

Signed-off-by: katelyn martin <[email protected]>

* refactor(app/outbound): type aliases in backend metrics layer

before we introduce additional layers to our backend stack, we should
define type aliases as prophylaxis against:
<https://rust-lang.github.io/rust-clippy/stable/index.html#type_complexity>

Signed-off-by: katelyn martin <[email protected]>

---------

Signed-off-by: katelyn martin <[email protected]>
kflynn pushed a commit to linkerd/website that referenced this pull request Nov 20, 2025
* feat(proxy-metrics): document outbound policy routing metrics

the documentation of our proxy metrics has not kept pace with all of the
exciting telemetry that has been introduced since
#1599 documented the state of our
authorization policy metrics.

this commit reworks the documentation to exhaustively document the
families of metrics exported by the proxy. this commit does not
introduce mention of the _inbound_ metrics that have been added, but
does rename this section to "_Endpoint Metrics_" in order to be
compatible with the future addition of `inbound_http_route_*`,
`inbound_http_route_backend_*`, `inbound_grpc_route*`, and
`inbound_grpc_route_backend_*` metrics.

see:
* linkerd/linkerd2-proxy#2377
* linkerd/linkerd2-proxy#2380
* linkerd/linkerd2-proxy#3086
* linkerd/linkerd2-proxy#3308
* linkerd/linkerd2-proxy#3334

Signed-off-by: katelyn martin <[email protected]>

* chore(markdownlint): allow duplicate "labels" headers

Signed-off-by: katelyn martin <[email protected]>

---------

Signed-off-by: katelyn martin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants