Skip to content

[processor/redaction] apply redaction to log.body (#37239) #37369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
May 7, 2025

Conversation

karakayasemi
Copy link
Contributor

@karakayasemi karakayasemi commented Jan 21, 2025

Description

This update applies redaction rules to the log.body field by supporting multiple data types, including string, slices, maps, and nested maps. It applies key-based configurations consistently across all map structures.

  • Key-Based Configurations: Applies redaction rules at all nesting levels.
  • New Debug Attributes:
    • redaction.body.redacted.keys / redaction.body.redacted.count
    • redaction.body.masked.keys / redaction.body.masked.count
    • redaction.body.allowed.keys / redaction.body.allowed.count
    • redaction.body.ignored.count

Link to tracking issue

Fixes #37239

Testing & Refactoring

  • Extended Tests: Log body redaction added to all existing cases + new tests for detailed body structure validation.
  • Code Cleanup: Introduced helper methods for redaction, masking, allowing, and ignoring logic.
  • Improved Readability: Renamed some const variables for clarity and consistency.

Copy link

linux-foundation-easycla bot commented Jan 21, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: karakayasemi / name: Semih Serhat Karakaya (4ebc42c, 0e184c8)
  • ✅ login: atoulme / name: Antoine Toulme (149f4ae)

@github-actions github-actions bot added the processor/redaction Redaction processor label Jan 21, 2025
@karakayasemi karakayasemi force-pushed the redact-log-body branch 2 times, most recently from 8c9b0ca to 3db3e73 Compare January 21, 2025 12:27
@karakayasemi
Copy link
Contributor Author

@dmitryax @mx-psi @TylerHelmuth @jpkrohling Any comment on the approach? I tried to keep the implementation as basic as possible. I can add a changelog and cover it with the tests if it seems good to go.

Copy link
Contributor

github-actions bot commented Feb 8, 2025

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Feb 8, 2025
Copy link
Member

@TylerHelmuth TylerHelmuth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@karakayasemi please add some tests for this new functionality

@github-actions github-actions bot removed the Stale label Feb 11, 2025
@karakayasemi karakayasemi force-pushed the redact-log-body branch 4 times, most recently from 3afc5d2 to 41f94cb Compare February 14, 2025 17:45
@karakayasemi
Copy link
Contributor Author

@TylerHelmuth I've added corresponding tests for string body redaction. When the body type is a map, I'm using the existing processAttrs method, which is already well-tested. Could you please take a look?

@gray-oni
Copy link

hey @karakayasemi the current implementation covers the situation when the Log's body is a string or one-level map value. However, if users use highly structured logs with nested map values (I'm one of these users🙂) it will not work (nested properties will be ignored and not redacted/masked). should we take into account the use-case of nested-structured logs body within the implementation? (otherwise this feature feels half-baked)

@karakayasemi
Copy link
Contributor Author

Hey @gray-oni,
Thanks for your feedback and for taking an interest in the PR! 🙌

The current implementation processes map values similarly to attribute handling—it converts them to a string, applies redaction, and then sets the modified value back. You can check the implementation here.

From this perspective, it should work as expected for now. However, I agree that supporting nested map structures would be a good improvement. But, IMHO, since this would impact logs, metrics, and traces, I believe it would be better to track it as a separate issue rather than expand the scope of this PR.

What do you think?

Copy link
Contributor

github-actions bot commented Mar 3, 2025

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Mar 3, 2025
@karakayasemi karakayasemi force-pushed the redact-log-body branch 2 times, most recently from 8ca8458 to 531b24a Compare March 7, 2025 14:17
@karakayasemi karakayasemi requested a review from dmitryax March 7, 2025 14:39
@karakayasemi karakayasemi force-pushed the redact-log-body branch 2 times, most recently from 8eb1482 to f9bf49a Compare March 7, 2025 14:45
@karakayasemi
Copy link
Contributor Author

karakayasemi commented Mar 25, 2025

@mx-psi @TylerHelmuth @dmitryax @evan-bradley @jpkrohling — Anything I can do to get attention on this PR? Appreciate your review when you get a chance!

Copy link
Contributor

github-actions bot commented Apr 9, 2025

This PR was marked stale due to lack of activity. It will be closed in 14 days.

@github-actions github-actions bot added the Stale label Apr 9, 2025
@mx-psi
Copy link
Member

mx-psi commented Apr 9, 2025

@dmitryax Since you started reviewing this, would you mind taking another look?

@atoulme
Copy link
Contributor

atoulme commented May 6, 2025

Passes my review. @mx-psi @TylerHelmuth @dmitryax please review as codeowners.

Copy link
Member

@dmitryax dmitryax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delayed response. The PR LGTM

@dmitryax dmitryax merged commit 9f2f46b into open-telemetry:main May 7, 2025
173 checks passed
@github-actions github-actions bot added this to the next release milestone May 7, 2025
@karakayasemi karakayasemi deleted the redact-log-body branch May 17, 2025 18:54
dragonlord93 pushed a commit to dragonlord93/opentelemetry-collector-contrib that referenced this pull request May 23, 2025
…) (open-telemetry#37369)

## Description  
This update applies redaction rules to the `log.body` field by
supporting multiple data types, including string, slices, maps, and
nested maps. It applies key-based configurations consistently across all
map structures.

- **Key-Based Configurations**: Applies redaction rules at all nesting
levels.
- **New Debug Attributes**:  
  - `redaction.body.redacted.keys` / `redaction.body.redacted.count`  
  - `redaction.body.masked.keys` / `redaction.body.masked.count`  
  - `redaction.body.allowed.keys` / `redaction.body.allowed.count`  
  - `redaction.body.ignored.count`  

#### Link to tracking issue
Fixes open-telemetry#37239

## Testing & Refactoring  
- **Extended Tests**: Log body redaction added to all existing cases +
new tests for detailed body structure validation.
- **Code Cleanup**: Introduced helper methods for redaction, masking,
allowing, and ignoring logic.
- **Improved Readability**: Renamed some const variables for clarity and
consistency.

---------

Co-authored-by: Antoine Toulme <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[processor/redaction] Add Support for Redacting log.body in the Redaction Processor
8 participants