redaction processor does not seem to redact attributes of span event

[qrli]

Component(s)

processor/redaction

What happened?

Description

We are using allow_all_keys: true and expects all blocked values are redacted. However, we found some data which should have been redacted got through. After investigation, I found the data failed to be redacted are in span events.

Steps to Reproduce

Put some value to be redacted in attributes of span events, and try to use redaction processor to redact it.

Expected Result

The blocked values are successfully redacted from all traces/logs.

Actual Result

The blocked values in span events leaked through redaction processor.

Collector version

0.111.0

Environment information

Environment

OS: Ubuntu 22.04
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

exporters:    
    otlp:
      endpoint: tempo.xxx.svc:4317
      sending_queue:
        queue_size: 5000
      tls:
        insecure: true
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  processors:
    redaction:
      allow_all_keys: true
      blocked_values:
        - "eyJhbGci[a-zA-Z0-9._-]+"  # jwt tokens
        - "(amqps|mongodb)://[^@]+@" # mq & mongo connection string
        - '[\w\.\-]+(@|%40)[\w\.\-]+' # email
      summary: debug
  service:
    pipelines:
      traces:
        exporters:
          - otlp
        receivers:
          - otlp
        processors: 
          - redaction

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• processor/redaction: @dmitryax @mx-psi @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dmitryax]

Hi @qrli. Thanks for reporting! It's clearly a bug. Do you have a chance contribute a fix?

[qrli]

Hi @qrli. Thanks for reporting! It's clearly a bug. Do you have a chance contribute a fix?

I'd like to but unfortunately Go is not in my skill set.

[VihasMakwana]

I'll find some time and fix this later this week. In the meantime, if anyone else wants to dive into the otel work, just let me know and I can assign it to them.

[VihasMakwana]

@qrli I tried to reproduce this with a couple of different kv pairs, here's my obervation:

• For email and jwt based tokens, the regex is able to capture entire string and it works as expected.
• For amqps\mongodb URIs, the regex is able to partially capture the string and as a result it doesn't mask the entire string.
  • Is this the issue you're facing?
  • If not, can you send a sample log and expected output?

[qrli]

@VihasMakwana No, the issue is not about my regex for redaction - I just provided them for example.
The issue is about the redaction does not apply to event attributes of spans.
I can see from the code here:

opentelemetry-collector-contrib/processor/redactionprocessor/processor.go

Line 95 in 464969b

s.processAttrs(ctx, spanAttrs)

It loops through all attributes of a span, but it does not touch any span events, so event attributes are not redacted.

So I think the fix should be simply adding a nested loop of span events and process each event attribute.

[qrli]

The issue is still marked as waiting for author. Is there anything more you need me to provide?

[mx-psi]

I think we are not waiting for author anymore, we just need to land the fix :)

[VihasMakwana]

I was working on the fix and then was on holidays for new yeas. Sorry for inconvenience.

I'll open up a PR soon.