subscriber: Per-layer filtering

[hawkw]

Feature Request

Crates

• tracing-subscriber

Motivation

A primary use-case for tracing-subscriber's Layer trait is to support teeing trace events between multiple consumers that implement different behaviors. When Layer is used in this way, it should be possible to add filtering to Layers to allow the individual Layer to decide what spans and events it is interested in recording, without disabling those spans/events for the entire stack.

Currently, however, the register_callsite and enabled methods on Layer are designed to support a use-case where a layer does deny or accept events for the whole stack. This is intended to allow a Layer to globally disable spans and events to support a use-case where a single filter is used for the whole stack, or use-cases such as rate-limiting etc.

This should probably be revisited. Ideally, it would be possible to support both use-cases without adding too much more additional complexity to the interface.

Design Questions

There are some design questions that should inform a solution to this problem.

• Are users interested in global filtering/sampling? The current design makes the assumption that users of tracing-subscriber want to be able to add a single filter/sampler layer to a stack that controls the behaviour of the entire composed subscriber. Is this actually the case? In a scenario where multiple layers that implement different ways of recording trace events (e.g. logging, metrics, etc) or output to different sinks (e.g., logging to a file, stderr, and stdout), will users want to have a global filter configuration that applies to all of these? And if so, is this common enough that we should optimize for it?
• Should Layer::enabled be used for per-layer filtering? Or, should a new trait method be added? Having two similar but different methods sounds potentially confusing.
• Are we okay with Layer::enabled being called multiple times for the same event?
• How to reconcile this with callsite Interest caching?

[hawkw]

@LucioFranco, @davidbarsky, @jarrodldavis it would be great to get your input on this if you're interested in weighing in.

I can flesh out the design questions a bit more but wanted to get the basic ideas written down first.

[jarrodldavis]

My main use-case would be separate filtering rules for separate sinks (likely via multiple subscribers à la #135).

[davidbarsky]

Here are my initial thoughts:

Are users interested in global filtering/sampling? The current design makes the assumption that users of tracing-subscriber want to be able to add a single filter/sampler layer to a stack that controls the behaviour of the entire composed subscriber. Is this actually the case? In a scenario where multiple layers that implement different ways of recording trace events (e.g. logging, metrics, etc) or output to different sinks (e.g., logging to a file, stderr, and stdout), will users want to have a global filter configuration that applies to all of these? And if so, is this common enough that we should optimize for it?

I think that a system like tracing—providing single, centralized instrumentation—is pretty novel. Therefore, we have the ability to set customer expectations around tracing and systems like it from the get-go. If my assumption is wrong, this can absolutely be revisited. To answer:

In a scenario where multiple layers that implement different ways of recording trace events (e.g. logging, metrics, etc) or output to different sinks (e.g., logging to a file, stderr, and stdout), will users want to have a global filter configuration that applies to all of these?

I think that the entire reason for splitting apart instrumentation data into constructs like metrics and logs is to enable necessary cost optimizations pertaining to ingestion volume and storage. For those use-cases, I think that per-layer filtering/selection is critical.

And if so, is this common enough that we should optimize for it?

Depending on what you mean by "optimize", but yes, I think that tracing customers will want this functionality from the start. As much as I want them to, they're not all going to move to a Honeycomb-style instrumentation system 😛.

Should Layer::enabled be used for per-layer filtering? Or, should a new trait method be added? Having two similar but different methods sounds potentially confusing.

Given the tree-like structure that emerges once Layers are composed, Layer::enabled feels like the correct solution because each Layer can selectively enable/disable layers downstream of it.

Are we okay with Layer::enabled being called multiple times for the same event?

I'm not sure this can be avoided in the case of multiple, disjoint Layers. However, for non-disjoint Layers—where the answer to Layer::enabled is the same—I think we can attempt to optimize this access if profiling indicates that this is an issue.

How to reconcile this with callsite Interest caching?

To connect back to the tree model of layers and subscribers, I view callsite Interest as an indication that at least one Layer is interested in this information. The fact that some layers choose to opt out of some events/spans is a desirable and acceptable usage of this API. This is what, I think, the Registry + Layer should enable customers to do with tracing.

[cpcloud]

Chiming in here as we've moved all of our logging to tracing and have recently added OpenTelemetry + Honeycomb into the mix.

Our main use case is to be able to send DEBUG (or possibly TRACE) level data to our OpenTelemetry collector (ultimately making its way into Honeycomb), while still preserving the much-less-verbose INFO level logging that our applications typically produce that goes to stdout.

We generally don't want the verbosity of DEBUG showing up in stdout, but we do want the high cardinality data that DEBUG provides going into the OpenTelemetry collector.

Looking at #508, the API seems solid. Attaching the filters directly to the layers seems like the right approach.

In this scheme, do global filters take precedence over layer-specific filters?

[cpcloud]

I'll try answering some of the design questions as they pertain to our use case:

Are users interested in global filtering/sampling?

I can't speak for everyone of course but IMO as soon as there's more than one layer the need for filtering on a per-layer basis becomes important. In our case, it's basically impossible for someone to make sense of DEBUG level traces as they are happening, e.g., by watching stdout, but that log spew is actually critical for a tool like Honeycomb so that you can query unaggregated data at the level of specific events (and also aggregate it in an ad-hoc fashion).

Should Layer::enabled be used for per-layer filtering?

From our perspective, we've yet to encounter having to call this method at all, which IMO is a win from the UX side so I don't really have any opinions here. It sounds like this is more relevant for Layer authors rather than the typical user of tracing. That said, I'm +1 on anything that makes extensibility less confusing.

Are we okay with Layer::enabled being called multiple times for the same event?
How to reconcile this with callsite Interest caching?

These two questions are beyond my current understanding of tracing :)

[cangSDARM]

Is there any new news about this issue?
What I saw seems to prove that the officially released version does not yet support layer-level filtering.