[Flaky Test] x-pack/metricbeat/tests/integration TestMetricbeatOTelMultipleReceiversE2E and TestFilebeatOTelMultipleReceiversE2E

[mauri870]

Flaky Test

• Test Name: TestMetricbeatOTelMultipleReceiversE2E
• Link:

  beats/x-pack/metricbeat/tests/integration/otel_test.go

  Line 428 in 73f8a2b

  require.NoError(t, err)

• Branch: main
• Artifact Link: https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/8e53406e-f6c4-8200-a234-e52417deb87c?period=1day
• Notes: This test passed the CI checks in the original PR, but it has a high failure rate in subsequent runs. It was introduced in otel: add mbreceiver E2E test #45427.

From the error message it seems to be an error returned by Elasticsearch, but I'm pretty sure this is a problem with two receivers on the same process, likely a global state issue somewhere. This is reinforced by the fact that the debugexporter only printed a single log, implying that the second receiver didn't even start.. I'll do a deeper investigation and post the results here.

Stack Trace

=== RUN   TestMetricbeatOTelMultipleReceiversE2E
    otel_test.go:428: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1753810346968905340/elastic/beats-xpack-metricbeat/x-pack/metricbeat/tests/integration/otel_test.go:428
        	            				/opt/buildkite-agent/.asdf/installs/golang/1.24.4/go/src/runtime/asm_arm64.s:1223
        	Error:      	Received unexpected error:
        	            	error in HTTP query: non-200 return code: 503, response: '[503 Service Unavailable] {
        	            	  "error" : {
        	            	    "root_cause" : [
        	            	      {
        	            	        "type" : "no_shard_available_action_exception",
        	            	        "reason" : "[e042caa97a13][127.0.0.1:9300][indices:data/read/search[phase/query]]"
        	            	      }
        	            	    ],
        	            	    "type" : "search_phase_execution_exception",
        	            	    "reason" : "all shards failed",
        	            	    "phase" : "query",
        	            	    "grouped" : true,
        	            	    "failed_shards" : [
        	            	      {
        	            	        "shard" : 0,
        	            	        "index" : ".ds-logs-integration-0169693e52434ebda3ee7b2ff1760d36-2025.07.29-000001",
        	            	        "node" : "KR2_yrWbTP6n41nvRjVFXA",
        	            	        "reason" : {
        	            	          "type" : "no_shard_available_action_exception",
        	            	          "reason" : "[e042caa97a13][127.0.0.1:9300][indices:data/read/search[phase/query]]"
        	            	        }
        	            	      }
        	            	    ]
        	            	  },
        	            	  "status" : 503
        	            	}
        	            	'
        	Test:       	TestMetricbeatOTelMultipleReceiversE2E
    otel_test.go:416: 
        	Error Trace:	/opt/buildkite-agent/builds/bk-agent-prod-aws-1753810346968905340/elastic/beats-xpack-metricbeat/x-pack/metricbeat/tests/integration/otel_test.go:416
        	Error:      	Condition never satisfied
        	Test:       	TestMetricbeatOTelMultipleReceiversE2E
        	Messages:   	expected at least 1 log for each receiver

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

Suddenly now we have more mbreceiver tests flaking:

TestMetricbeatOTelReceiverE2E https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/475e47dc-b872-899a-bb85-1b7a8aa34b06?branch=main&period=1day

TestMetricbeatOTelE2E https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/e1031fe7-bff1-86e8-9217-b9523a8925a6?branch=main&period=1day

[mauri870]

After investigating this error locally, here is my findings. On my local machine, after bringing up es with mage docker:composeUp the instance has 1000 shards available.

After updating the test framework to print more lines on failure, I found the following elasticsearchexporter log buried under the metricbeat logs:

Meta: none' (status=400): {\"type\":\"validation_exception\",\"reason\":\"Validatio
n Failed: 1: this action would add [2] shards, but this cluster currently has [1000]/[1000] maximum normal shards open; for more information, see [https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;\](https://www.elastic.co/guide/en/elasticsearch/reference/8.17/size-your-shards.html#troubleshooting-max-shards-open;%5C)"}, dropping event!","[service.name":"metricbeat","log.type":"event","ecs.version":"1.6.0](about:blank)"}

Apparently, each index requires 2 shards.

I noticed that we don't clean up indexes after each test, so it could be that we are hitting the shard limit because of it.

[mauri870]

Even after cleaning up indexes and starting a fresh elasticsearch container it still fails on the same error:

$ go test -tags "integration" -run 'TestMetricbeatOTel.*' ./tests/integration -v -count=100
=== RUN   TestMetricbeatOTelE2E
    otel_test.go:133:
                Error Trace:    /home/mauri870/git/elastic/beats/x-pack/metricbeat/tests/integration/otel_test.go:133
                                                        /usr/lib/go/src/runtime/asm_amd64.s:1700
                Error:          Received unexpected error:
                                error in HTTP query: non-200 return code: 503, response: '[503 Service Unavailable] {
                                  "error" : {
                                    "root_cause" : [
                                      {
                                        "type" : "no_shard_available_action_exception",
                                        "reason" : null
                                      }
                                    ],
                                    "type" : "search_phase_execution_exception",
                                    "reason" : "all shards failed",
                                    "phase" : "query",
                                    "grouped" : true,
                                    "failed_shards" : [
                                      {
                                        "shard" : 0,
                                        "index" : ".ds-logs-integration-mb-6f86ca73490b4e9685b379ab0953c6f9-2025.08.04-000001",
                                        "node" : null,
                                        "reason" : {
                                          "type" : "no_shard_available_action_exception",
                                          "reason" : null
                                        }
                                      }
                                    ]
                                  },
                                  "status" : 503
                                }
                                '
                Test:           TestMetricbeatOTelE2E

I double checked and the index count is 14, we can't be hitting the 1000 shard limit with indexes alone.

Invoking the cluster allocation explain I get:

curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" | jq
      ...
      "deciders": [
        {
          "decider": "same_shard",
          "decision": "NO",
          "explanation": "a copy of this shard is already allocated to this node [[.ds-logs-integration-mbreceiver-5bdb1d2bed73488087b67673f61c7853-2025.08.04-000001][0], node[uOmnf37RTMayHbGko_FWxQ], [P], s[STARTED], a[id=3unKVFEyRH2pQjtRxytcVw], failed_attempts[0]]"
        }
      ...

[mauri870]

Also tried to remove the restriction of one shard per node, still failing with the same error and a very small number of indexes:

$ curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '
{
  "persistent": {
    "cluster.max_shards_per_node": 10000,
    "cluster.max_shards_per_node.frozen": 10000,
    "cluster.routing.allocation.enable": "all",
    "cluster.routing.allocation.allow_rebalance": "always",
    "cluster.routing.allocation.same_shard.host": true
  }
}'

[belimawr]

I've just encountered the same error on TestFilebeatOTelMultipleReceiversE2E

    otel_test.go:584: 
                Error Trace:    /home/tiago/devel/beats/x-pack/filebeat/tests/integration/otel_test.go:584
                                                        /home/tiago/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.4.linux-amd64/src/runtime/asm_amd64.s:1700
                Error:          Received unexpected error:
                                error in HTTP query: non-200 return code: 503, response: '[503 Service Unavailable] {
                                  "error" : {
                                    "root_cause" : [
                                      {
                                        "type" : "no_shard_available_action_exception",
                                        "reason" : null
                                      }
                                    ],
                                    "type" : "search_phase_execution_exception",
                                    "reason" : "all shards failed",
                                    "phase" : "query",
                                    "grouped" : true,
                                    "failed_shards" : [
                                      {
                                        "shard" : 0,
                                        "index" : ".ds-logs-integration-ef5fe6aa44504424a1d066e5c4d2d56c-2025.08.07-000001",
                                        "node" : null,
                                        "reason" : {
                                          "type" : "no_shard_available_action_exception",
                                          "reason" : null
                                        }
                                      }
                                    ]
                                  },
                                  "status" : 503
                                }
                                '
                Test:           TestFilebeatOTelMultipleReceiversE2E
    otel_test.go:578: 
                Error Trace:    /home/tiago/devel/beats/x-pack/filebeat/tests/integration/otel_test.go:578
                Error:          Condition never satisfied
                Test:           TestFilebeatOTelMultipleReceiversE2E
                Messages:       expected at least 200 events, got 0
    framework.go:293: CMD line to execute Beat: /home/tiago/devel/beats/x-pack/filebeat/filebeat.test filebeat-otel --systemTest --path.home /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763 --path.logs /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763 -E logging.to_files=true -E logging.files.rotateeverybytes=104857600 -E logging.files.rotateonstartup=false otel -c /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/filebeat-otel.yml
    otel_test.go:562: Config contents:
        receivers:
        
          filebeatreceiver/0:
            filebeat:
              inputs:
                - type: filestream
                  id: filestream-fbreceiver
                  enabled: true
                  paths:
                    - /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/log.log
                  prospector.scanner.fingerprint.enabled: false
                  file_identity.native: ~
            output:
              otelconsumer:
            logging:
              level: info
              selectors:
                - '*'
            queue.mem.flush.timeout: 0s
            path.home: /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/r1
        
            http.enabled: true
            http.host: localhost
            http.port: 5066
        
        
          filebeatreceiver/1:
            filebeat:
              inputs:
                - type: filestream
                  id: filestream-fbreceiver
                  enabled: true
                  paths:
                    - /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/log.log
                  prospector.scanner.fingerprint.enabled: false
                  file_identity.native: ~
            output:
              otelconsumer:
            logging:
              level: info
              selectors:
                - '*'
            queue.mem.flush.timeout: 0s
            path.home: /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/r2
        
            http.enabled: true
            http.host: localhost
            http.port: 5067
        
        
        exporters:
          debug:
            use_internal_logger: false
            verbosity: detailed
          elasticsearch/log:
            endpoints:
              - http://localhost:9200
            compression: none
            user: admin
            password: testing
            logs_index: logs-integration-ef5fe6aa44504424a1d066e5c4d2d56c
            batcher:
              enabled: true
              flush_timeout: 1s
            mapping:
              mode: bodymap
        service:
          pipelines:
            logs:
              receivers:
        
                - filebeatreceiver/0
        
                - filebeatreceiver/1
        
              exporters:
                - debug
                - elasticsearch/log
    framework.go:1046: Last 2048 bytes of stderr:
        ol.signal": "logs", "service.name": "filebeat", "id": "filestream-fbreceiver", "ecs.version": "1.6.0"}
        2025-08-07T11:30:19.043-0400    info    crawler beater/crawler.go:182   Crawler stopped {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}, "otelcol.component.id": "filebeatreceiver/1", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"}
        2025-08-07T11:30:19.043-0400    info    registrar       registrar/registrar.go:126      Stopping Registrar      {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}, "otelcol.component.id": "filebeatreceiver/1", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"}
        2025-08-07T11:30:19.043-0400    info    registrar       registrar/registrar.go:162      Ending Registrar        {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}, "otelcol.component.id": "filebeatreceiver/1", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"}
        2025-08-07T11:30:19.043-0400    info    registrar       registrar/registrar.go:131      Registrar stopped       {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}, "otelcol.component.id": "filebeatreceiver/1", "otelcol.component.kind": "receiver", "otelcol.signal": "logs", "service.name": "filebeat", "ecs.version": "1.6.0"}
        2025-08-07T11:30:19.051-0400    info    extensions/extensions.go:69     Stopping extensions...  {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}}
        2025-08-07T11:30:19.051-0400    info    service@v0.130.0/service.go:336 Shutdown complete.      {"resource": {"service.instance.id": "bc671879-2b12-4fdc-8ae2-5e0645975706", "service.name": "otel", "service.version": "9.2.0"}}
    framework.go:1052: Last 2048 bytes of stdout:
        ersion":""}},"input":{"type":"filestream"},"log":{"file":{"device_id":"64768","inode":"13541828","path":"/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/log.log"},"offset":774},"message":"Line 98"})
        Trace ID: 
        Span ID: 
        Flags: 0
        LogRecord #68
        ObservedTimestamp: 2025-08-07 15:29:52.114706629 +0000 UTC
        Timestamp: 2025-08-07 15:29:52.114706629 +0000 UTC
        SeverityText: 
        SeverityNumber: Unspecified(0)
        Body: Map({"@timestamp":"2025-08-07T15:29:52.114Z","agent":{"ephemeral_id":"460366cc-0b26-43b2-a7e3-3a4b522f9cef","id":"e37cae1b-b076-4cbd-b9fe-218f3866a753","name":"millennium-falcon","type":"filebeat","version":"9.2.0"},"container":{"id":"x-pack"},"ecs":{"version":"8.0.0"},"host":{"architecture":"x86_64","containerized":false,"hostname":"millennium-falcon","id":"851f339d77174301b29e417ecb2ec6a8","ip":["192.168.42.173","fdeb:9303:5e1f::61f","fdeb:9303:5e1f:0:8aa0:ddda:5b47:22a3","fe80::d4ee:e28:a936:8282","100.84.12.120","fd7a:115c:a1e0:ab12:4843:cd96:6254:c78","fe80::861d:2a46:5dd4:8bad","10.84.78.1","172.17.0.1","fe80::5049:acff:fef4:6de8","172.18.0.1","fe80::8060:a5ff:fe64:c464","fe80::98c1:54ff:fe5d:f1d2","fe80::30fa:11ff:feb5:493","fe80::8cc4:f2ff:fe51:9988","fe80::b891:cbff:fedc:cdd3","fe80::88c0:64ff:fe3f:a53a"],"mac":["0A-00-27-00-00-00","0A-00-27-00-00-01","0A-00-27-00-00-02","0A-00-27-00-00-03","0A-00-27-00-00-04","32-FA-11-B5-04-93","52-49-AC-F4-6D-E8","52-54-00-EA-A0-7B","82-60-A5-64-C4-64","8A-C0-64-3F-A5-3A","8E-92-06-A7-57-26","8E-C4-F2-51-99-88","9A-C1-54-5D-F1-D2","BA-91-CB-DC-CD-D3","C8-A3-62-C5-A5-7F"],"name":"millennium-falcon","os":{"build":"rolling","family":"arch","kernel":"6.15.9-arch1-1","name":"Arch Linux","platform":"arch","type":"linux","version":""}},"input":{"type":"filestream"},"log":{"file":{"device_id":"64768","inode":"13541828","path":"/home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763/log.log"},"offset":782},"message":"Line 99"})                                                  
        Trace ID: 
        Span ID: 
        Flags: 0
        
        PASS
    framework.go:740: Temporary directory saved: /home/tiago/devel/beats/x-pack/filebeat/build/integration-tests/TestFilebeatOTelMultipleReceiversE2E1211641763
--- FAIL: TestFilebeatOTelMultipleReceiversE2E (30.27s)

[belimawr]

It's failing on main and some PRs as well.