[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

djaglowski · 2025-04-08T19:08:38Z

Depends on #12856

Resolves #12676

This is a reboot of #11311, incorporating metrics defined in the component telemetry RFC and attributes added in #12617.

The basic pattern is:

When building any pipeline component which produces data, wrap the "next consumer" with instrumentation to measure the number of items being passed. This wrapped consumer is then passed into the constructor of the component.
When building any pipeline component which consumes data, wrap the component itself. This wrapped consumer is saved onto the graph node so that it can be retrieved during graph assembly.

codecov · 2025-04-08T19:30:13Z

Codecov Report

Attention: Patch coverage is 82.98507% with 57 lines in your changes missing coverage. Please review.

Project coverage is 91.54%. Comparing base (ac520a5) to head (7c557ad).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
service/internal/graph/connector.go	65.95%	32 Missing and 16 partials ⚠️
service/internal/graph/exporter.go	85.71%	2 Missing and 1 partial ⚠️
service/internal/graph/processor.go	89.65%	2 Missing and 1 partial ⚠️
service/internal/graph/receiver.go	75.00%	2 Missing and 1 partial ⚠️

❌ Your patch status has failed because the patch coverage (82.98%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12812      +/-   ##
==========================================
- Coverage   91.67%   91.54%   -0.13%     
==========================================
  Files         503      503              
  Lines       27819    28094     +275     
==========================================
+ Hits        25502    25720     +218     
- Misses       1828     1866      +38     
- Partials      489      508      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jaronoff97

some initial thoughts, overall implementation seems very sane to me though. Thank you!

service/internal/graph/connector.go

service/internal/graph/exporter.go

service/metadata.yaml

service/documentation.md

docs/rfcs/component-universal-telemetry.md

service/metadata.yaml

codeboten

Thanks for moving this work forward @djaglowski, just a few questions

service/internal/graph/receiver.go

djaglowski · 2025-04-11T16:25:16Z

Thanks for the reviews.

#12817 implements a subset of this PR. If we can get that merged in first, I'll rebase to reduce the scope of this one substantially.

MikeGoldsmith

I think this looks really good 👍🏻

As shared in the other PR, I'd really like to see bytes counters too but this is a great start.

Subset of #12812 This internal package defines wrappers around consumers. These are useful for instrumenting the component graph, so that we can generate telemetry describing data as it is passed in between components. Currently, this supports only a single counter metric, but in the near future it can be enhanced to automatically capture multiple metrics (e.g. item count & size), and potentially spans and/or logs as well.

config/configtls/go.mod

service/internal/graph/obs_test.go

Implements new name for attribute as documented in #12951. Note: The obsconsumer package is unused until #12812 is merged so this is not a breaking change.

… of go

mx-psi · 2025-05-07T15:49:19Z

@dmitryax @codeboten I will merge this by ~~Friday EOD~~ Monday morning EU time unless you block by then

codeboten

Thanks @djaglowski, just one comment about the update of the tpm package, otherwise this looks good

config/configtls/go.mod

MikeGoldsmith

🎉

Follows #12812 This PR adds the `size` metrics defined in the [Pipeline Component Telemetry RFC](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md).

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [go.opentelemetry.io/collector/component](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v1.31.0` -> `v1.32.0` | | [go.opentelemetry.io/collector/component/componenttest](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v0.125.0` -> `v0.126.0` | | [go.opentelemetry.io/collector/confmap](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v1.31.0` -> `v1.32.0` | | [go.opentelemetry.io/collector/consumer](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v1.31.0` -> `v1.32.0` | | [go.opentelemetry.io/collector/consumer/consumertest](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v0.125.0` -> `v0.126.0` | | [go.opentelemetry.io/collector/pdata](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v1.31.0` -> `v1.32.0` | | [go.opentelemetry.io/collector/processor](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v1.31.0` -> `v1.32.0` | | [go.opentelemetry.io/collector/processor/processortest](https://github.com/open-telemetry/opentelemetry-collector) | require | minor | `v0.125.0` -> `v0.126.0` | --- ### Release Notes <details> <summary>open-telemetry/opentelemetry-collector (go.opentelemetry.io/collector/component)</summary> ### [`v1.32.0`](https://github.com/open-telemetry/opentelemetry-collector/blob/HEAD/CHANGELOG.md#v1320v01260) ##### 🛑 Breaking changes 🛑 - `configauth`: Removes deprecated `configauth.Authentication` and `extensionauthtest.NewErrorClient` ([#12992](open-telemetry/opentelemetry-collector#12992)) The following have been removed: - `configauth.Authentication` use `configauth.Config` instead - `extensionauthtest.NewErrorClient` use `extensionauthtest.NewErr` instead ##### 💡 Enhancements 💡 - `service`: Replace `go.opentelemetry.io/collector/semconv` usage with `go.opentelemetry.io/otel/semconv` ([#12991](open-telemetry/opentelemetry-collector#12991)) - `confmap`: Update the behavior of the confmap.enableMergeAppendOption feature gate to merge only component lists. ([#12926](open-telemetry/opentelemetry-collector#12926)) - `service`: Add item count metrics defined in Pipeline Component Telemetry RFC ([#12812](open-telemetry/opentelemetry-collector#12812)) See [Pipeline Component Telemetry RFC](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/rfcs/component-universal-telemetry.md) for more details: - `otelcol.receiver.produced.items` - `otelcol.processor.consumed.items` - `otelcol.processor.produced.items` - `otelcol.connector.consumed.items` - `otelcol.connector.produced.items` - `otelcol.exporter.consumed.items` - `tls`: Add trusted platform module (TPM) support to TLS authentication. ([#12801](open-telemetry/opentelemetry-collector#12801)) Now the TLS allows the use of TPM for loading private keys (e.g. in TSS2 format). ##### 🧰 Bug fixes 🧰 - `exporterhelper`: Add validation error for batch config if min_size is greater than queue_size. ([#12948](open-telemetry/opentelemetry-collector#12948)) - `telemetry`: Allocate less memory per component when OTLP exporting of logs is disabled ([#13014](open-telemetry/opentelemetry-collector#13014)) - `confmap`: Use reflect.DeepEqual to avoid panic when confmap.enableMergeAppendOption feature gate is enabled. ([#12932](open-telemetry/opentelemetry-collector#12932)) - `internal telemetry`: Add resource attributes from telemetry.resource to the logger ([#12582](open-telemetry/opentelemetry-collector#12582)) Resource attributes from telemetry.resource were not added to the internal console logs. Now, they are added to the logger as part of the "resource" field. - `confighttp and configcompression`: Fix handling of `snappy` content-encoding in a backwards-compatible way ([#10584](open-telemetry/opentelemetry-collector#10584), [#12825](open-telemetry/opentelemetry-collector#12825)) The collector used the Snappy compression type of "framed" to handle the HTTP content-encoding "snappy". However, this encoding is typically used to indicate the "block" compression variant of "snappy". This change allows the collector to: - When receiving a request with encoding 'snappy', the server endpoints will peek at the first bytes of the payload to determine if it is "framed" or "block" snappy, and will decompress accordingly. This is a backwards-compatible change. If the feature-gate "confighttp.framedSnappy" is enabled, you'll see new behavior for both client and server: - Client compression type "snappy" will now compress to the "block" variant of snappy instead of "framed". Client compression type "x-snappy-framed" will now compress to the "framed" variant of snappy. - Servers will accept both "snappy" and "x-snappy-framed" as valid content-encodings. - `tlsconfig`: Disable TPM tests on MacOS/Darwin ([#12964](open-telemetry/opentelemetry-collector#12964))  </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://gitea.t000-n.de/t.behrendt/tracebasedlogsampler/pulls/13 Co-authored-by: Renovate Bot <[email protected]> Co-committed-by: Renovate Bot <[email protected]>

github-actions bot requested review from bogdandrutu, codeboten, dmitryax and mx-psi April 8, 2025 19:09

djaglowski changed the title ~~Compiling and tests passing~~ [service/internal/graph] Record normalized telemetry as it is passed between pipeline components Apr 8, 2025

djaglowski changed the title ~~[service/internal/graph] Record normalized telemetry as it is passed between pipeline components~~ [service/internal/graph] Measure telemetry as it is passed between pipeline components Apr 8, 2025

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from b6bb02d to 2f83f2b Compare April 8, 2025 19:20

jaronoff97 reviewed Apr 8, 2025

View reviewed changes

service/internal/graph/connector.go Outdated Show resolved Hide resolved

service/internal/graph/connector.go Show resolved Hide resolved

service/internal/graph/exporter.go Show resolved Hide resolved

service/metadata.yaml Show resolved Hide resolved

djaglowski force-pushed the pipeline-component-metrics branch from 2f83f2b to 052cffc Compare April 8, 2025 20:36

mx-psi reviewed Apr 9, 2025

View reviewed changes

service/documentation.md Outdated Show resolved Hide resolved

djaglowski commented Apr 9, 2025

View reviewed changes

docs/rfcs/component-universal-telemetry.md Outdated Show resolved Hide resolved

djaglowski force-pushed the pipeline-component-metrics branch from 052cffc to 362a610 Compare April 9, 2025 16:39

djaglowski mentioned this pull request Apr 9, 2025

[chore] Create service/internal/obsconsumer package #12817

Merged

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from fbed573 to b789109 Compare April 9, 2025 20:24

codeboten reviewed Apr 11, 2025

View reviewed changes

service/metadata.yaml Show resolved Hide resolved

service/metadata.yaml Show resolved Hide resolved

codeboten reviewed Apr 11, 2025

View reviewed changes

service/internal/graph/receiver.go Outdated Show resolved Hide resolved

MikeGoldsmith approved these changes Apr 14, 2025

View reviewed changes

djaglowski force-pushed the pipeline-component-metrics branch from b789109 to 4afe6cd Compare April 15, 2025 13:13

djaglowski mentioned this pull request Apr 15, 2025

Permanently enable 'telemetry.newPipelineTelemetry' feature gate #12856

Merged

djaglowski force-pushed the pipeline-component-metrics branch from 4afe6cd to 5bdd398 Compare April 15, 2025 14:29

github-actions bot requested a review from dmathieu April 15, 2025 14:29

djaglowski force-pushed the pipeline-component-metrics branch 4 times, most recently from adec160 to b66a5a8 Compare April 17, 2025 14:47

djaglowski force-pushed the pipeline-component-metrics branch 3 times, most recently from 07a92cf to b54f142 Compare April 30, 2025 18:06

djaglowski marked this pull request as ready for review April 30, 2025 18:57

djaglowski commented Apr 30, 2025

View reviewed changes

config/configtls/go.mod Outdated Show resolved Hide resolved

djaglowski force-pushed the pipeline-component-metrics branch from b54f142 to 3b4dc35 Compare May 2, 2025 14:07

djaglowski mentioned this pull request May 2, 2025

[chore] Update obsconsumer to use new attribute name #12959

Merged

mx-psi reviewed May 5, 2025

View reviewed changes

service/internal/graph/obs_test.go Show resolved Hide resolved

mx-psi approved these changes May 5, 2025

View reviewed changes

djaglowski requested review from codeboten and dmitryax May 6, 2025 14:53

djaglowski added 2 commits May 7, 2025 07:39

[chore] Update recently added dependency to fix lint on some versions…

2870909

… of go

Add consumer and producer item count metrics

488b658

djaglowski force-pushed the pipeline-component-metrics branch 2 times, most recently from 3398805 to c5e2fcf Compare May 7, 2025 14:55

codeboten approved these changes May 7, 2025

View reviewed changes

config/configtls/go.mod Outdated Show resolved Hide resolved

dmitryax approved these changes May 7, 2025

View reviewed changes

Enabled telemetry based on featuregate

014f717

djaglowski force-pushed the pipeline-component-metrics branch from e25097a to 014f717 Compare May 7, 2025 23:54

Merge branch 'main' into pipeline-component-metrics

7c557ad

MikeGoldsmith approved these changes May 12, 2025

View reviewed changes

mx-psi added this pull request to the merge queue May 12, 2025

Merged via the queue into open-telemetry:main with commit 279752c May 12, 2025
55 of 56 checks passed

djaglowski mentioned this pull request May 14, 2025

[service/internal/graph] Add size throughput metrics #13032

Merged

vigneshshanmugam mentioned this pull request May 19, 2025

[Ratelimit Processor] Instrument the ratelimiter service with telemetry metrics elastic/opentelemetry-collector-components#562

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

Uh oh!

djaglowski commented Apr 8, 2025 •

edited

Loading

Uh oh!

codecov bot commented Apr 8, 2025 •

edited

Loading

Uh oh!

jaronoff97 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codeboten left a comment

Uh oh!

Uh oh!

djaglowski commented Apr 11, 2025

Uh oh!

MikeGoldsmith left a comment

Uh oh!

Uh oh!

Uh oh!

mx-psi commented May 7, 2025 •

edited

Loading

Uh oh!

codeboten left a comment

Uh oh!

Uh oh!

MikeGoldsmith left a comment

Uh oh!

Uh oh!

Uh oh!

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

[service/internal/graph] Measure telemetry as it is passed between pipeline components #12812

Uh oh!

Conversation

djaglowski commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jaronoff97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codeboten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

djaglowski commented Apr 11, 2025

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mx-psi commented May 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codeboten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MikeGoldsmith left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

djaglowski commented Apr 8, 2025 •

edited

Loading

codecov bot commented Apr 8, 2025 •

edited

Loading

mx-psi commented May 7, 2025 •

edited

Loading