Attribute injection in the Collector

[jade-guiton-dd]

Problem

The Collector SIG is working on various improvements to the Collector's internal telemetry, among which is an RFC specifying attributes used to identify which Collector component emitted a given item of telemetry (eg. "the transform/foo processor in pipeline metrics/bar").

We are looking for ways to attach these "component attributes" to internal Collector telemetry automatically, without needing component developers to manually add them. (See the discussion in this issue.)

Current solutions

This has proven to be complicated with the current OTel API and data model:

• resource attributes are set once when creating a Provider, and cannot be changed without recreating the telemetry exporting pipeline, whereas we need different values for different components;
• instrumentation scope attributes have the wrong semantics (they identify parts of the code, not runtime component instances), and are poorly supported (if at all) by observability backends;
• span/metric/log attributes are normally set in the final call, inside component code which we don't control.

Our current best solution (in this PR) uses span/metric/log attributes, and involves wrapping the TracerProvider and MeterProvider we pass to the component, as well as the Tracer, Meter, and all instruments created from them, to thread the attributes through the component and inject them in the final call. This works, but is brittle and inelegant.

Another proposed idea is to thread the attributes through the Context passed to components, and inject the attributes using SDK processors. However, this would require some cooperation on the component side (carefully threading Contexts and never using context.Background), and because metric processors do not exist, would still require an unwieldy wrapper-based workaround.

Question

Does the Go SIG know of any current or planned SDK-level or API-level features we could leverage to make this easier to accomplish?

Here are some ideas we thought of which would fit our use case:

• the ability to define default span/metric/log attributes on Tracers/Meters/Loggers
• the ability to specialize a Provider with additional resource attributes, without recreating the exporting pipeline
• the ability to define "metric processors", akin to span/log processors
  (note that there is an ongoing specification effort related to this)
  • perhaps additionally, the ability to define extra Baggage associated with a Meter/Tracer/Logger, to avoid issues with threading Contexts through components

If you have ideas for alternative solutions to this using purely the OTel API, we are of course also interested.

Additional context

Possibly related use cases:

• There are features in specific instrumentation libraries in go-contrib which allow adding arbitrary attributes to the emitted telemetry: in otelhttp, in otelgrpc, work in progress
• This feature request in the Collector could probably be solved with metric processors

[jmacd]

@jade-guiton-dd Thank you, this is great. I would like to propose to elevate this to the specification level, as I do not believe the stated problems are specific to the OTel Collector. You've listed three solutions and all of them have been discussed in the past and yet we are stalled.

Here are some brief positions of mine:

instrumentation scope attributes have the wrong semantics (they identify parts of the code, not runtime component instances), and are poorly supported (if at all) by observability backends;

I think this is an unfortunate situation, and I would focus our efforts on resolving this. IMO InstrumentationScope is the correct place for these attributes to go; while the name and version refer to a body of source code, the attributes field was originally intended to do what you're looking for. It's OpenTelemetry's problem that we've had a scope attributes field for years and still can't use it. Let's fix this.

I think this is the best approach compared with the others you mentioned. Our goal should be to avoid repetition of attributes for each span/metric/log because it is an important optimization.

span/metric/log attributes are normally set in the final call, inside component code which we don't control.

OpenTelemetry has (for years) talked about a specific solution in metric instrumentation called the "Bound" instrument, which is a bound instrument for metric-ing with pre-specified attributes. Many metrics and logging APIs outside OTel allow you to register attributes that are automatically inserted to every log or metric (e.g. zap's Logger.With(), Prometheus' MetricVec.CurryWith()), and still there's no way to do that in OpenTelemetry. While I think OTel should move in this direction, I would still prefer to see component-level attributes as stated in open-telemetry/opentelemetry-collector#12217 use instrumentation-scope attributes to avoid repetition.

Another proposed idea is to thread the attributes through the Context passed to components, and inject the attributes using SDK processors. However, this would require some cooperation on the component side (carefully threading Contexts and never using context.Background), and because metric processors do not exist, would still require an unwieldy wrapper-based workaround.

See the discussion in open-telemetry/oteps#207. Again, this is something I think OpenTelemetry should work out. You linked to the work on MetricsProcessor, which goes part of the way to a solution: OTEP 207 takes it further by proposing that OTel provide a standard way to enter context that would automatically be extracted by the SDKs. I would like to see such functionality in the Collector, but I would not use it for component-level attributes. The place I would use this in a Collector, for example, is to attach information propagated in-band that is not component-specific, like information about the end-user who submitted the request or the identity of the downstream destination calculated by an upstream load balancer. (Also yes, Go developers should always be careful about correct use of context.)

[mx-psi]

@jmacd

It's OpenTelemetry's problem that we've had a scope attributes field for years and still can't use it. Let's fix this.

It's unclear to me how we fix this. Do you have a concrete proposal on how to go about it?

[jmacd]

@mx-psi This will require a group discussion at the specification level to make progress. My proposal is to push forward with the Scope Attributes concept. Here's the original OTEP:

https://github.com/open-telemetry/opentelemetry-specification/blob/main/oteps/0201-scope-attributes.md

and the associated issue:

open-telemetry/opentelemetry-specification#2450

My understanding is that there are conflicting opinions about how attributes should be interpreted on Scopes and when it is safe for SDKs to permit multiple scopes. The topic is stalled, and I can't fully answer why in brief. If you submit OTLP data with "identifying" scope attributes, they need to be elevated to metric-level attributes by Prometheus, for example. If Prometheus exporter doesn't know about Scope attributes, the feature is a loss; if the Prometheus exporter includes non-identifying attributes, the feature is a breaking change.

Still, I think Scope Attributes are the answer the problem. When the Collector's TelemetrySettings object is being constructed, each component's object name, pipeline name, signal type and so on will be included as scope attributes. The trick will be reaching an agreement on how this can be done without unnecessary harm. E.g.,

TelemetrySettings{
  Tracer: provider.NewTracer("qualified/name", WithVersion("0.125.0"), WithAttributes(attribute.String("exporter", "otlp"), attribute.String("pipeline", "traces/named"), attribute.String("signal", "traces")),
}

I base this proposal on the assertion that right here, OpenTelemetry has discovered within itself a shortcoming where instrumentation is hard to get right (and efficient). Let's figure out the best approach to instrumentation here, and the standardize it across OpenTelemetry SDKs.

[jade-guiton-dd]

@jmacd Do you think I should create a new issue on the specification repo to discuss whether scope attributes are the right way to go on this?

[jmacd]

This is on the discussion agenda for 3/11/2025 in the Specification SIG.
After the discussion, we may consider moving this issue into the specification repo.

[jade-guiton-dd]

My takeaways from the Spec SIG meeting: (@jmacd feel free to add anything relevant I missed)

• The idea of creating a different resource per component was rejected.
• While scopes are sometimes narrowly understood to correspond strictly to compile-time code boundaries, no language in the spec (outside of one instance in the glossary) requires this. As long as they remain static over the lifetime of the component, our component attributes would fit in scope attributes perfectly fine.
• We should not worry too much about whether backend vendors support scope attributes. Adopting them now should not be too difficult, and would benefit use cases outside of the Collector's.
• Not much discussion of bound Loggers/Tracers/Meters; only a brief mention of metric processors.

[axw]

While scopes are sometimes narrowly understood to correspond strictly to compile-time code boundaries, no language in the spec (outside of one instance in the glossary) requires this. As long as they remain static over the lifetime of the component, our component attributes would fit in scope attributes perfectly fine.

@jmacd @jade-guiton-dd would it be worth updating the spec to make it more explicitly allowed, rather than being left to interpretation? I have only ever seen examples that refer to static scopes, e.g. "module, package, or class" at https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope and https://opentelemetry.io/docs/concepts/instrumentation-scope/

[jade-guiton-dd]

@axw Definitely, yes. Your first link is the glossary entry I mentioned above, which is technically under "specs" and is unnecessarily restrictive at the moment. The second link should be non-normative, but I agree that adding an explicit example of a runtime scope would be beneficial.

[jmacd]

Thank you @jade-guiton-dd. I agree with your summary, and I'm glad that scope attributes is the outcome. I agree that OpenTelemetry should look at its documentation about this topic, too. I would suggest renaming this issue, maybe to "Endorse use of Scoped Attributes in instrumentation" and transferring the issue to the specification repository.

[jade-guiton-dd]

This issue has a lot of info that I don't think will be very relevant for rewording the spec; I'll create a new issue on the spec repo and close this one.

[jade-guiton-dd]

Spec issue created here: open-telemetry/opentelemetry-specification#4450