-
Notifications
You must be signed in to change notification settings - Fork 2.6k
Fix data race in ESv1 WriteSpan by using distinct inputs #7196
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
@yurishkuro I would like your review, Also please let me know if i need to write any test |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Under what conditions is the same span being modified? Storage should not be in the business of modifying input data, it has internal db model it can use to modify.
c084740
to
d6e1546
Compare
Under what conditions is the same span being modified? Storage should not be in the business of modifying input data, it has internal db model it can use to modify. |
To fix this, now I create a deep copy before applying any changes, so the input span remains untouched.. |
You are not answering my question. The fix should be not mutating the data, not making a copy of the data and then mutating it. Copying is expensive at scale. |
I realize that the real issue is mutating the input span, which ideally should be treated as immutable by the storage layer. Allow me some time, and I will fix it. |
@yurishkuro Can you give me a little direction on whether I should apply transform without mutation? |
@sonalgaud12 I don't think we should be fixing the problem without understanding the root cause. First, let's find the code in storage that mutates input data. Then decide if that mutation is actually necessary or if the same capability can be achieved without it. |
ffb212b
to
ff9e69b
Compare
@sonalgaud12 Please see the comment #7168 (comment). Most probably it will fix the issue. |
Signed-off-by: sonalgaud12 <[email protected]>
Signed-off-by: sonalgaud12 <[email protected]>
Signed-off-by: Sonal Gaud <[email protected]> Signed-off-by: sonalgaud12 <[email protected]>
Signed-off-by: sonalgaud12 <[email protected]>
Signed-off-by: sonalgaud12 <[email protected]>
Signed-off-by: sonalgaud12 <[email protected]>
d7ff8a6
to
b46a236
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This fixes the symptom, not the root cause. The storage MUST NOT mutate input data. In the context of OTEL collector framework, it's perfectly valid to configure multiple exporters for the same data, and if one of the exporters mutates this data it would still cause race conditions.
But it isn't the model or otel span, it's the db span. Shouldn't that work? |
@Manik2708 ah, good point |
Also another point, it is not related to V1, but to the base factory. I think we should now work on moving the shared code from v1 to v2 so that in future no v1 dependent code is added in v2. And it could easily be removed in future. |
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #7196 +/- ##
=======================================
Coverage 96.17% 96.17%
=======================================
Files 366 366
Lines 21947 21947
=======================================
Hits 21107 21107
Misses 627 627
Partials 213 213
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
@yurishkuro Hi, all checks have passed. Could you please re-add this to the merge queue or merge it? Thanks! |
Which problem is this PR solving?
Description of the changes
How was this change tested?
Checklist
jaeger
:make lint test
jaeger-ui
:npm run lint
andnpm run test