Skip to content

[filelog] Can't remove syslog parts from log messages - collector crashes #40398

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
driprado opened this issue Jun 2, 2025 · 3 comments
Open

[filelog] Can't remove syslog parts from log messages - collector crashes #40398

driprado opened this issue Jun 2, 2025 · 3 comments
Labels
question Further information is requested receiver/filelog waiting-for-code-owners

Comments

@driprado
Copy link

driprado commented Jun 2, 2025

Component(s)

receiver/filelog

What happened?

Description

When using the remove operator after syslog_parser operator in filelog receiver, the collector panics with a nil pointer dereference. The goal is to remove specific syslog fields (PRI, VERSION, MSGID, STRUCTURED-DATA) from the output, but any attempt to use the remove operator results in a panic.

Steps to Reproduce

  1. Configure filelog receiver with syslog_parser operator
  2. Add remove operator to remove syslog fields
  3. Start the collector
  
  Configuration:
  ```yaml
  receivers:
    filelog/my-app-name:
      include_file_name: false
      resource:
        service.name: my-app-name
      storage: file_storage/filelog
      include:
      - "/var/log/folder/my-app-name/standard.log"
      operators:
      - id: syslog-parser
        type: syslog_parser
        protocol: rfc5424
      - id: remove-fields
        type: remove
        fields:
          - priority
          - version
          - msgid
          - structured_data
      retry_on_failure:
        enabled: true
  ```

Expected Result

  The remove operator should successfully remove specified syslog fields from the log output. For example, given this input:
  
  <139>1 2025-05-30T10:52:12+10:00 localhost my-app-name 1761 - -  #011at java.base/java.lang.Thread.run(Thread.java:840)
  
  It should output only the remaining fields:
  
  2025-05-30T10:52:12+10:00 localhost my-app-name 1761 #011at java.base/java.lang.Thread.run(Thread.java:840)

Actual Result

collector crashes with error:

May 30 11:26:56  stdout-docker_otel[32833]: panic: runtime error: invalid memory address or nil pointer dereference
May 30 11:26:56  stdout-docker_otel[32833]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x3c307ad]
May 30 11:26:56  stdout-docker_otel[32833]: goroutine 106 [running]:
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/entry.(*Entry).Delete(...)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/entry/entry.go:63
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/remove.(*Transformer).Transform(0xc0008c1520, 0x0?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/transformer/remove/transformer.go:41 +0x8d
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessWith(0xc0008c1520, {0x5695528, 0xc0005e9cc0}, 0xc000740e40, 0xc000ac5800)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/transformer.go:100 +0x48
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/transformer/remove.(*Transformer).Process(0x4e37760?, {0x5695528?, 0xc0005e9cc0?}, 0xc000acd590?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/transformer/remove/transformer.go:26 +0x35
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).Write(0xc0004cb400, {0x5695528, 0xc0005e9cc0}, 0xc000740e40)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/writer.go:73 +0x236
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*ParserOperator).ProcessWithCallback(0xc0004cb400, {0x5695528, 0xc0005e9cc0}, 0xc000740e40, 0xc000a3e988, 0x50b7620)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/parser.go:125 +0x10d
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).Process(0xc000f186b0?, {0x5695528?, 0xc0005e9cc0?}, 0x3c3b1e5?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/parser/syslog/parser.go:58 +0x148
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*TransformerOperator).ProcessBatchWith(0xc000a3ea68?, {0x5695528, 0xc0005e9cc0}, {0xc000042708, 0x64, 0xc000a3ea68?}, 0xc000ac5a68)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/transformer.go:84 +0x58
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/parser/syslog.(*Parser).ProcessBatch(0x0?, {0x5695528?, 0xc0005e9cc0?}, {0xc000042708?, 0x0?, 0x0?})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/parser/syslog/parser.go:41 +0x3a
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/helper.(*WriterOperator).WriteBatch(0xc000e55a50, {0x5695528, 0xc0005e9cc0}, {0xc000042708, 0x64, 0x64})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/helper/writer.go:55 +0x294
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/operator/input/file.(*Input).emitBatch(0xc000e55a40, {0x5695528, 0xc0005e9cc0}, {0xc000ade008?, 0x95?, 0xc000ad1fcc?}, 0x95?, 0x2034?)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/operator/input/file/input.go:49 +0xf2
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).readContents(0xc000e691d0, {0x5695528, 0xc0005e9cc0})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/internal/reader/reader.go:235 +0x429
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer/internal/reader.(*Reader).ReadToEnd(0xc000e691d0, {0x5695528, 0xc0005e9cc0})
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/internal/reader/reader.go:117 +0x7e9
May 30 11:26:56  stdout-docker_otel[32833]: github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume.func1(0xc000e691d0)
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/file.go:169 +0xa5
May 30 11:26:56  stdout-docker_otel[32833]: created by github.com/open-telemetry/opentelemetry-collector-contrib/pkg/stanza/fileconsumer.(*Manager).consume in goroutine 57
May 30 11:26:56  stdout-docker_otel[32833]:         github.com/open-telemetry/opentelemetry-collector-contrib/pkg/[email protected]/fileconsumer/file.go:166 +0x205
May 30 11:26:56  systemd[1]: docker-otel.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 30 11:26:56  systemd[1]: docker-otel.service: Unit entered failed state.
May 30 11:26:56  systemd[1]: docker-otel.service: Failed with result 'exit-code'.

Collector version

v0.113.0

Environment information

Environment

x86_64 x86_64 x86_64 GNU/Linux

OpenTelemetry Collector configuration

---
receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-fields
      type: remove
      fields:
      - attributes.procid
      - attributes.msgid
      - attributes.structured_data
    retry_on_failure:
      enabled: true
processors:
  batch: {}
  memory_limiter:
    check_interval: 1s
    limit_mib: 100
  resource/loki:
    attributes:
    - action: insert
      key: loki.format
      value: raw
  attributes:
    actions:
    - key: source
      value: on-premise
      action: insert
    - key: env
      value: stg
      action: insert
    - key: host
      value: server-01
      action: insert
    - key: datacenter
      value: dc1
      action: insert
  attributes/loki:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: stream, source, env, host, datacenter, level
  attributes/stream_stdout:
    actions:
    - key: stream
      value: stdout
      action: insert
  deltatocumulative: {}
exporters:
  otlphttp/logs:
    logs_endpoint: https://logs.logging.stg.example.cloud/otlp/v1/logs
    sending_queue:
      storage: file_storage/queue
  loki:
    endpoint: https://logs.logging.stg.example.cloud/loki/api/v1/push
    sending_queue:
      storage: file_storage/queue
service:
  extensions:
  - health_check
  - file_storage/queue
  - file_storage/filelog
  telemetry:
    metrics:
      readers:
      - pull:
          exporter:
            prometheus:
              host: 0.0.0.0
              port: 9777
  pipelines:
    logs/my-app-1:
      receivers:
      - filelog/my-app-1
      processors:
      - memory_limiter
      - filter
      - batch
      - resource/loki
      - attributes/loki
      - attributes/stream_stdout
      - attributes
      exporters:
      - loki
      - count
extensions:
  health_check:
    endpoint: 0.0.0.0:13133
  file_storage/queue:
    directory: "/file_storage"
  file_storage/filelog:
    directory: "/file_storage"
    compaction:
      directory: "/file_storage/compact"
connectors:
  count: {}

Log output



Additional context


  - Have tried using both `field` and `fields` in remove operator
  - Have tried with `attributes.` prefix for field names
  - Have tried removing single fields instead of multiple
  - Other receivers in the same config work correctly
  - The syslog_parser operator works correctly on its own


      		
    

      
  





        



            
            

              
            

        

      


      
    








      

    



      

  
            

    

      
    

    


      

          @driprado
driprado




            added
      

  bug

  Something isn't working

      

  needs triage

  New item requiring triage

  labels


      Jun 2, 2025

    

  



    







      

  
      



  

  @github-actions

    
      GitHub Actions
    




  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
    Contributor


      

  


  

    

      

      
            github-actions
  bot

      

      

      commented


        Jun 2, 2025


      
    


  





      


        


  
    

      
          
Pinging code owners:



See Adding Labels via Comments if you do not have permissions to add labels yourself.

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
      



  

  @andrzej-stencel





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      

  
  Member


      

  


  

    

  





      


        


  
    

      
          
The remove operator does not have a fields property, it has and always had a singular field property.


To remove multiple fields, define multiple instances of the operator, like this:


receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-procid
      type: remove
      field: attributes.procid
    - id: remove-msgid
      type: remove
      field: attributes.msgid
    - id: remove-structured-data
      type: remove
      field: attributes.structured_data
    retry_on_failure:
      enabled: true


The collector will log errors like the following when the field to be removed does not exist in a log:


2025-06-02T15:19:06.908+0200    error   helper/transformer.go:118       Failed to process entry {"resource": {}, "otelcol.component.id": "filelog", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "operator_id": "remove-procid", "operator_type": "remove", "error": "remove: field does not exist: attributes.procid", "action": "send", "entry.timestamp": "0001-01-01T00:00:00.000Z"}


To prevent this, set the operators' on_error property to send_quiet.


Having said all this, the collector should not panic when misconfigured; it should rather fail to start.


Let me know if this works for you @driprado.

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
            

    

      
    

    


      

          @andrzej-stencel
andrzej-stencel




            added
      

  waiting for author


    and removed
      

  needs triage

  New item requiring triage

  labels


      Jun 2, 2025

    

  



  

  

    
  


  

      

    @github-actions
github-actions
bot


    mentioned this issue
    
      Jun 3, 2025
    





  


  

      
  
  Open

  







  









      

  
      



  

  @driprado





  


    

      

  

    

        
    
    

  


      
          
  
      
            Copy link

  

      
    

  


  

      



      

  Author


  


  

    

      

      
            driprado
  

      

      

      commented


        Jun 6, 2025


      
    


  





      


        


  
    

      
          
Thank you for your response @andrzej-stencel


I have configured the operator as instructed:


receivers:
  filelog/my-app-1:
    include_file_name: false
    resource:
      service.name: my-app-1
    storage: file_storage/filelog
    include:
    - "/var/log/apps/my-app-1/standard.log"
    operators:
    - id: syslog-parser
      type: syslog_parser
      protocol: rfc5424
    - id: remove-msgid
      type: remove
      field: attributes.msgid
    retry_on_failure:
      enabled: true



And the collector started with no errors, however there were no changes to log output, and the msgid field has not been removed:


<139>1 2025-06-06T11:21:02+10:00 localhost my-app-1 1761 - -  #011at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

      		
    

  





        


            

            
            

              
            

        

      


      

            
  
  

    
  
    
    

  

  

  


    












      

  
            

    

      
    

    


      

          @andrzej-stencel
andrzej-stencel




            added
      

  question

  Further information is requested

      

  waiting-for-code-owners


    and removed
      

  bug

  Something isn't working

      

  waiting for author


  labels


      Jun 6, 2025

    

  












  
  

    

      

  





  




  
  

              

  
    Sign up for free
    to join this conversation on GitHub.
    Already have an account?
    Sign in to comment


  



  






  
                  




    

  


      
  

    Assignees
  



        

    No one assigned







      


  


  

    Labels
  



    

      

    question

  Further information is requested
  

    receiver/filelog

  

    waiting-for-code-owners









      

  

    

        

    Projects
  


        




    None yet




  



      


  

    
  

    Milestone
  


      No milestone





    
      
          


  

    

      
        

          
    

    Development
  




            
    
No branches or pull requests







      
    

  




      

    

  
      

  

    

      2 participants
    

    

        
          @driprado 
        
          @andrzej-stencel