How to do regex matching in router expr

[kaushal02]

I have the following otel collector config:

receivers:
  filelog/nginx/1:
    include:
    - /opt/nginx/log/access.log
    include_file_path: true
    include_file_name: false
    operators:
    - type: router
      routes:
        - output: parse1
          expr: 'body matches "^\\[\\d"'
          attributes:
            debug_body_router: '${body}'
      default: parse2
    - type: regex_parser
      id: parse1
        # Example line:
        # [28/Jan/2025:04:32:25 +0000] 127.0.0.1 34096 /__cq/status ...
      regex: '^\[(?P<timestamp>\d{2}\/[A-Za-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+\-]\d{4})\]\s+(?P<ip>\S+)\s+(?P<port>\d+)\
s+(?P<path>\S+).*$'
      timestamp:
        parse_from: attributes.timestamp
        layout: '%d/%b/%Y:%H:%M:%S %z'
      attributes:
        debug_body_parse1: '${body}'
    - type: regex_parser
      id: parse2
        # Example lines:
        # 127.0.0.1 - - [29/Jan/2025:08:33:22 +0000] ...
        # 127.0.0.1 - MitigatorConnector [28/Jan/2025:14:23:46 +0000] ...
      regex: '^(?P<ip>\S+)\s+-\s+(?P<service>\S+)\s+\[(?P<timestamp>\d{2}\/[A-Za-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+\-]\d
{4})\]\s+(?P<msg>.*)$'
      timestamp:
        parse_from: attributes.timestamp
        layout: '%d/%b/%Y:%H:%M:%S %z'
      attributes:
        debug_body_parse2: '${body}'

exporters:
  debug:
    verbosity: "detailed"

service:
  pipelines:
    logs:
      receivers: [filelog/nginx/1]
      processors: []
      exporters: [debug]
  telemetry:
    logs:
      level: debug

I see the following in collector log:

LogRecord #1
ObservedTimestamp: 2025-01-30 09:15:19.792630808 +0000 UTC
Timestamp: 2025-01-30 09:15:19 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str([30/Jan/2025:09:15:19 +0000] 127.0.0.1 37158 /__cq/cache/info /__cq/cache/info 200 214 "python-requests/2.25.1" 0.000 "-" "-" "-" "-" "14183" "167" "-" "localhost:9999" - -)
Attributes:
     -> path: Str(/__cq/cache/info)
     -> log.file.path: Str(/opt/nginx/log/access.log)
     -> debug_body_router: Str()
     -> timestamp: Str(30/Jan/2025:09:15:19 +0000)
     -> ip: Str(127.0.0.1)
     -> port: Str(37158)
Trace ID: 
Span ID: 

And there are parse2 and router "regex pattern does not match" errors. I have few concerns:

• why am I seeing regex pattern does not match error for router? Isn't that the whole point of routers? If it does not match, it should move to the next route instead of throwing an error.
• if the regex pattern did not match for the router, why do I still see the attribute debug_body_router in the exported log?
• most importantly, how do I match the body correctly in the router expr?

[kaushal02]

As a workaround, I was able to get the desired behavior by using if and removing router altogether. Updated receiver config for reference:

receivers:
  filelog/nginx/1:
    include:
    - /opt/nginx/log/access.log
    include_file_path: true
    include_file_name: false
    operators:
    - type: regex_parser
      id: parse1
        # Example line:
        # [28/Jan/2025:04:32:25 +0000] 127.0.0.1 34096 /__cq/status ...
      if: 'body matches "^\\[\\d"'
      regex: '^\[(?P<timestamp>\d{2}\/[A-Za-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+\-]\d{4})\]\s+(?P<ip>\S+)\s+(?P<port>\d+)\s+(?P<path>\S+).*$'
      timestamp:
        parse_from: attributes.timestamp
        layout: '%d/%b/%Y:%H:%M:%S %z'
    - type: regex_parser
      id: parse2
        # Example lines:
        # 127.0.0.1 - - [29/Jan/2025:08:33:22 +0000] ...
        # 127.0.0.1 - MitigatorConnector [28/Jan/2025:14:23:46 +0000] ...
      if: 'body not matches "^\\[\\d"'
      regex: '^(?P<ip>\S+)\s+-\s+(?P<service>\S+)\s+\[(?P<timestamp>\d{2}\/[A-Za-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2}\s+[+\-]\d{4})\]\s+(?P<msg>.*)$'
      timestamp:
        parse_from: attributes.timestamp
        layout: '%d/%b/%Y:%H:%M:%S %z'

[matthewbyrne]

Very similar scenario as yourself. I'm trying to use a generic regex_parser for all logs, pass that to a router where I use expressions to match log formats to further regex_parser operators via the output field. Even though my logs are parsed correctly, and stored correctly, I still get errors in the logs:

2025-04-28T14:08:03.258Z	error	router/transformer.go:82	Failed to process entry	{"operator_id": "route_logs", "operator_type": "router", "entry.timestamp": "0001-01-01T00:00:00.000Z", "log.file.name": "application.log", "full_log_line": "2025-04-28 15:02:23 INFO CPU usage: 95%", "error": "regex pattern does not match"}
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/router.(*Transformer).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/transformer/router/transformer.go:82
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/writer.go:73
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/parser.go:125
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/parser.go:98
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).Process
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/parser/regex/parser.go:35
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessBatchWith
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/transformer.go:82
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/regex.(*Parser).ProcessBatch
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/parser/regex/parser.go:30
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).WriteBatch
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/writer.go:55
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emitBatch
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/input/file/input.go:48
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).readContents
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/internal/reader/reader.go:232
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/internal/reader/reader.go:116
github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/file.go:169

otel config:

receivers:
  filelog:
    include:
      - /logs/*.log
    start_at: beginning
    operators:
      - type: router
        id: route_logs
        routes:
          # Standard format
          - output: [standard_logs]
            expr: 'body matches "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2} [A-Z]+ .*(\\n)?"'
            attributes:
              debug_router: 'standard'
          # Alternative format
          - output: [alt_logs]
            expr: 'body matches "^[A-Z]+ \\d{4}-\\d{2}-\\d{2} .*$"'
            attributes:
              debug_router: 'alt'
        default: [catchall]

      - type: regex_parser
        id: standard_logs
        regex: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?P<severity>\w+) (?P<log_message>.*)$'

      - type: regex_parser
        id: alt_logs
        regex: '^(?P<severity>\w+) (?P<time>\d{4}-\d{2}-\d{2}) (?P<log_message>.*)$'

      - type: regex_parser
        id: catchall
        regex: '(?P<log>.*)'