otel: add test for otelconsumer + elasticsearchexporter document-level retries

[mauri870]

Proposed commit message

This commit adds a test to verify that elasticsearchexporter applies
document-level retries correctly, and that the combination of
otelconsumer and the exporter guarantees at-least-once delivery when
a batch is acknowledged.

The test simulates retryable and permanent errors, checks that
max_retries is respected, and ensures that documents retried after
temporary failures are ingested while permanent failures are dropped.

Currenty, otelconsumer output metrics are not tracked correctly, and it
will always ACK the whole batch even if there were retries or dropped
events by the exporter.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

$ cd x-pack/filebeat
$ mage buildSystemTestBinary
$ go test -run ^TestFilebeatOTelDocumentLevelRetries$ -v -count=1 -tags=integration ./tests/integration

$ ./script/stresstest.sh --tags integration ./x-pack/filebeat/tests/integration ^TestFilebeatOTelDocumentLevelRetries$ -p 1
10m20s: 30 runs so far, 0 failures, 1 active
10m25s: 30 runs so far, 0 failures, 1 active

Related issues

• Depends on otel: handle compression_level 0 correctly in config translation #46391
• Depends on sending_queue from [beatreceiver] - Disable force_attempt_http2 and map ES config to elasticsearchexporter #46111
• For [beats receivers] The interactions between otelconsumer and the ES exporter lose data on partial failure of _bulk #45707

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

/test

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fbreceiver-mockes-test upstream/fbreceiver-mockes-test
git merge upstream/main
git push upstream fbreceiver-mockes-test

[mauri870]

/test

[mauri870]

This is ready for review now, pending CI.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fbreceiver-mockes-test upstream/fbreceiver-mockes-test
git merge upstream/main
git push upstream fbreceiver-mockes-test

[VihasMakwana]

LGTM, just a nit.

[mauri870]

TestFilestreamTruncateBlockedOutput is flaking https://github.com/elastic/ingest-dev/issues/6087.

Edit: Fix here #46576.

[mauri870]

@cmacknz When you have a chance, could you please review this and approve if it looks good? Thanks!

[github-actions]

@Mergifyio backport 9.0 9.1

[mergify]

backport 9.0 9.1

✅ Backports have been created

• #46617 [9.0](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries has been created for branch 9.0 but encountered conflicts
• #46619 [9.1](backport #46413) otel: add test for otelconsumer + elasticsearchexporter document-level retries has been created for branch 9.1 but encountered conflicts