Skip to content

Conversation

@mingley
Copy link

@mingley mingley commented Dec 17, 2025

Motivation

tracing_subscriber::reload::Layer wraps the inner Layer/Filter in an RwLock. In many common Tokio setups (typical async request handling on a runtime with a small, fixed number of worker threads), this does not usually show up as a problem: tasks are multiplexed onto a bounded number of OS threads and the extra read-side synchronization is often lost in the noise.

However, the reload layer is also used in environments where there is real OS-thread parallelism at high log volume (e.g., spawn_blocking, rayon thread pools, or other dedicated thread pools) and where spans/events are emitted frequently from many threads at once. In those cases, the read-side overhead of an RwLock can become a measurable and sometimes dominant cost, even when the filter is never reloaded.

This PR addresses that niche-but-real performance cliff while keeping the existing behavior as the default.

Refs: #2658

Solution

  • Add reload::ArcSwapLayer/reload::ArcSwapHandle behind a new reload-arc-swap feature.
    • Uses arc_swap::ArcSwap for a lock-free read path.
    • Serializes reload/modify operations with a Mutex (reloads are expected to be rare).
    • Calls tracing_core::callsite::rebuild_interest_cache() after updates (matching reload::Layer semantics), and updates log max-level when tracing-log is enabled.
    • Requires L: Clone for the Layer impl (because on_layer needs &mut L).

This is additive:

  • The existing reload::Layer remains unchanged.
  • arc-swap is an optional dependency.
  • Users who don’t enable the feature or don’t opt into ArcSwapLayer see no behavior or dependency changes.

Benchmark

A small Criterion benchmark was added to compare baseline vs reload::Layer vs reload::ArcSwapLayer:

  • cargo bench -p tracing-subscriber --bench reload --features reload-arc-swap

Note: the multi-threaded benchmarks are intentionally constructed to create OS-thread parallelism (via std::thread, tokio::spawn_blocking, and a Rayon pool) in order to exacerbate read-side synchronization contention. This is not representative of typical async request-handling on a small number of Tokio worker threads.

Results (Apple M4 Pro, 14 cores, 48GB RAM; macOS 26.2; rustc 1.92.0; Criterion point estimates; short run settings):

Benchmark Baseline (no reload) reload::Layer (RwLock) reload::ArcSwapLayer (ArcSwap)
single_threaded 4.88 ns 8.90 ns (1.82x) 9.58 ns (1.96x)
multithreaded_16x1000 (std::thread) 67.2 µs 11.9 ms (177x) 71.7 µs (1.07x)
tokio_spawn_blocking_16x1000 57.1 µs 12.8 ms (223x) 62.8 µs (1.10x)
rayon_16x1000 39.4 µs 15.0 ms (380x) 51.9 µs (1.32x)

(*_16x1000 means 16 workers/tasks; each emits 3 events (info/debug/trace) 1000x with an INFO filter.)

Testing

  • cargo test -p tracing-subscriber --all-features
  • cargo clippy -p tracing-subscriber --all-features -- -D warnings

Add an optional, lock-free alternative to reload::Layer using arc-swap.

This introduces reload::ArcSwapLayer and reload::ArcSwapHandle behind the

new reload-arc-swap feature. Reloads rebuild the callsite interest cache,

matching reload::Layer semantics.

A small Criterion benchmark is added to compare baseline vs reload::Layer

and reload::ArcSwapLayer.

Refs: tokio-rs#2658
@hds
Copy link
Contributor

hds commented Dec 17, 2025

Firstly, thanks for your PR!

With the current tracing maintainer team being stretched so thin at the moment. We're trying to avoid adding new features to the tracing crates where it isn't necessary.

Is there any reason why this improved reload layer couldn't be split out into its own crate? Since it depends on only the public API of some tracing crates, it doesn't need to be inside the tracing-subscriber crate. If it is split into a separate crate, then we'll happily add it to the Related Crates section of the README.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants