[processor/tailsampling] Record which sampling policy was responsible for the decision #37797
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jpkrohling
approved these changes
Feb 10, 2025
There was a problem hiding this comment.
LGTM, changing the existing benchmark to record the policy shows that the costs are OK:
This change (plus setting tsp.recordPolicy = true in BenchmarkSampling):
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkSampling$ github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkSampling-16 40365 28595 ns/op 6282 B/op 258 allocs/op
PASS
ok github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor 1.478s
Baseline:
Running tool: /usr/bin/go test -benchmem -run=^$ -bench ^BenchmarkSampling$ github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
goos: linux
goarch: amd64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
cpu: 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz
BenchmarkSampling-16 47973 24604 ns/op 6260 B/op 257 allocs/op
PASS
ok github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor 1.458s
jpkrohling
changed the title
|
@jpkrohling thanks for the review 🙇 |
|
Once the CI is green, I think this is ready to be merged. |
|
Apologies, I missed that check- just pushed a change after running |
…olicy) associated with an inclusive tail processor sampling decision. Resolves !35180. - This functionality lives behind a feature flag that is disabled by default - The original issue described a solution where we might attach the attribute solely to the root span. I'm not sure I agree with the commenter that we can rely on this (e.g. we might decide to sample halfway through a long-running trace) so I have attached the attributes to all present scope spans. This feels like a decent trade off between complexity + network cost, as finding the highest non-root parent would require multiple passes of the spans and keeping all span ids in a set - Added automated tests to verify enabling the flag both records the expected decision while not impacting existing logic - Built a custom version and ran it in our preprod environment to ensure it was stable over a 1h period (still evaluating, will update PR with any further observations) Does this require a CHANGELOG entry?
- Added README entry for the feature flag - Added missing mutex lock around reading trace data in `SetAttrOnScopeSpans` - Added tests + benchmarks for `SetAttrOnScopeSpans`
…olicy) associated with an inclusive tail processor sampling decision. Resolves !35180. - This functionality lives behind a feature flag that is disabled by default - The original issue described a solution where we might attach the attribute solely to the root span. I'm not sure I agree with the commenter that we can rely on this (e.g. we might decide to sample halfway through a long-running trace) so I have attached the attributes to all present scope spans. This feels like a decent trade off between complexity + network cost, as finding the highest non-root parent would require multiple passes of the spans and keeping all span ids in a set - Added automated tests to verify enabling the flag both records the expected decision while not impacting existing logic - Built a custom version and ran it in our preprod environment to ensure it was stable over a 1h period (still evaluating, will update PR with any further observations) Does this require a CHANGELOG entry?
djluck
force-pushed
the
tailsampling-record-policy
branch
from
913ab9c to
6a3b105
Compare
djluck
requested review from
dashpole,
MovieStoreGuy,
andrzej-stencel and
crobert-1
as code owners
|
@jpkrohling I re-ran |
|
@jpkrohling looks like we're green 🥳 Do you mind merging when you get a moment? |
|
Merged, thanks! |
|
Wonderful, thanks again for all your help @jpkrohling |
|
It looks like this PR conflicts with #37035 merged two days ago, which removed the Update: I opened #37931 to fix this. |
jade-guiton-dd
added a commit
to jade-guiton-dd/opentelemetry-collector-contrib
that referenced
this pull request
Feb 14, 2025
songy23
pushed a commit
that referenced
this pull request
Feb 14, 2025
#### Description Two PRs were merged recently on the tailsamplingprocessor, #37797 and #37035. #37035 changed the signature of an internal function in a way that broke #37797. The result is that the component [fails to build](https://github.com/open-telemetry/opentelemetry-collector/actions/runs/13329091378/job/37228871811?pr=12384). This PR fixes that. This wasn't noticed before merging because 1. there were no merge conflicts, 2. the latest rebase of #37797 was before #37035 was merged, and 3. there is no merge queue to perform final checks. --------- Signed-off-by: Juraci Paixão Kröhling <[email protected]> Co-authored-by: Juraci Paixão Kröhling <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Re-opening a stale PR: #36312
Resolves #35180.
We we're close to it being merged. @jpkrohling do you mind finalizing the review when you get a chance?