otel: fix retries in otelconsumer and add receivertest test suite #45637

mauri870 · 2025-07-30T14:37:22Z

Proposed commit message

It appears that the otelconsumer's Publish method returning an error prevented batches from being retried, despite the batch.Retry call. As a result, events with initial errors were never delivered. Changing the return value to nil fixed the issue.

Additionally, use OTel's receivertest.CheckConsumeContract to test filebeatreceiver. Some piping is required to forward the unique test ID attribute expected by receivertest.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

How to test this PR locally

$ go test -timeout 5m -tags integration,unit -run ^TestConsumeContract$ ./x-pack/filebeat/fbreceiver -v -count=1

=== RUN   TestConsumeContract
=== RUN   TestConsumeContract/always_succeed
Sent 100, accepted=100, expected dropped=0, non-permanent errors retried=0
=== RUN   TestConsumeContract/random_non_permanent_error
Sent 100, accepted=100, expected dropped=0, non-permanent errors retried=88
=== RUN   TestConsumeContract/random_permanent_error
Sent 100, accepted=50, expected dropped=50, non-permanent errors retried=0
=== RUN   TestConsumeContract/random_error
Sent 100, accepted=45, expected dropped=55, non-permanent errors retried=48
--- PASS: TestConsumeContract (12.11s)
    --- PASS: TestConsumeContract/always_succeed (3.03s)
    --- PASS: TestConsumeContract/random_non_permanent_error (3.03s)
    --- PASS: TestConsumeContract/random_permanent_error (3.03s)
    --- PASS: TestConsumeContract/random_error (3.03s)
PASS
ok  	github.com/elastic/beats/v7/x-pack/filebeat/fbreceiver	12.152s

./script/stresstest.sh --tags unit ./x-pack/filebeat/fbreceiver ^TestConsumeContract$
3m50s: 576 runs so far, 0 failures, 32 active

Related issues

Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest.

github-actions · 2025-07-30T14:37:34Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

mauri870 · 2025-07-30T16:31:08Z

/test

x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

elasticmachine · 2025-07-31T11:38:05Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

x-pack/filebeat/fbreceiver/receiver_test.go

x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

khushijain21

Thanks for your work on this!

leehinman

LGTM. Thank You.

pierrehilbert

Based on Lee's approval, I'm giving mine.

github-actions · 2025-08-06T16:21:53Z

@Mergifyio backport 9.0 9.1

mergify · 2025-08-06T16:22:02Z

backport 9.0 9.1

✅ Backports have been created

#45773 [9.0](backport #45637) otel: fix retries in otelconsumer and add receivertest test suite has been created for branch 9.0 but encountered conflicts
#45774 [9.1](backport #45637) otel: fix retries in otelconsumer and add receivertest test suite has been created for branch 9.1 but encountered conflicts

…5637) * otel: use receivertest to test fbreceiver Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest. * add a mutex since Generate runs in parallel * seems like returning an error in the output disables retries * go mod tidy * make notice * fix otelconsumer tests to asser no error * fix batch test * revert otelconsumer error message * let the testing pkg remove the file when the test ends * go mod tidy * add isReceiverTest property to otelconsumer * fix typo * add doc comment for TestConsumeContract * rollback changes to go.mod and go.sum * update NOTICE.txt (cherry picked from commit ee17a83) # Conflicts: # NOTICE.txt # go.mod # x-pack/filebeat/fbreceiver/receiver_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go

…5637) * otel: use receivertest to test fbreceiver Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest. * add a mutex since Generate runs in parallel * seems like returning an error in the output disables retries * go mod tidy * make notice * fix otelconsumer tests to asser no error * fix batch test * revert otelconsumer error message * let the testing pkg remove the file when the test ends * go mod tidy * add isReceiverTest property to otelconsumer * fix typo * add doc comment for TestConsumeContract * rollback changes to go.mod and go.sum * update NOTICE.txt (cherry picked from commit ee17a83) # Conflicts: # NOTICE.txt # go.mod

…ivertest test suite (#45774) * otel: fix retries in otelconsumer and add receivertest test suite (#45637) * otel: use receivertest to test fbreceiver Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest. * add a mutex since Generate runs in parallel * seems like returning an error in the output disables retries * go mod tidy * make notice * fix otelconsumer tests to asser no error * fix batch test * revert otelconsumer error message * let the testing pkg remove the file when the test ends * go mod tidy * add isReceiverTest property to otelconsumer * fix typo * add doc comment for TestConsumeContract * rollback changes to go.mod and go.sum * update NOTICE.txt (cherry picked from commit ee17a83) # Conflicts: # NOTICE.txt # go.mod * fix conflicts --------- Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>

…ivertest test suite (#45773) * otel: fix retries in otelconsumer and add receivertest test suite (#45637) * otel: use receivertest to test fbreceiver Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest. * add a mutex since Generate runs in parallel * seems like returning an error in the output disables retries * go mod tidy * make notice * fix otelconsumer tests to asser no error * fix batch test * revert otelconsumer error message * let the testing pkg remove the file when the test ends * go mod tidy * add isReceiverTest property to otelconsumer * fix typo * add doc comment for TestConsumeContract * rollback changes to go.mod and go.sum * update NOTICE.txt (cherry picked from commit ee17a83) # Conflicts: # NOTICE.txt # go.mod # x-pack/filebeat/fbreceiver/receiver_test.go # x-pack/libbeat/outputs/otelconsumer/otelconsumer.go * fix conflicts * remove unused code --------- Co-authored-by: Mauri de Souza Meneguzzo <[email protected]>

…astic#45637) * otel: use receivertest to test fbreceiver Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest. * add a mutex since Generate runs in parallel * seems like returning an error in the output disables retries * go mod tidy * make notice * fix otelconsumer tests to asser no error * fix batch test * revert otelconsumer error message * let the testing pkg remove the file when the test ends * go mod tidy * add isReceiverTest property to otelconsumer * fix typo * add doc comment for TestConsumeContract * rollback changes to go.mod and go.sum * update NOTICE.txt

mauri870 added 6 commits May 13, 2025 09:59

otel: use receivertest to test fbreceiver

6249c55

Use the receivertest.CheckConsumeContract to test filebeatreceiver. Requires some piping to forward the unique test id attribute required by receivertest.

Merge branch 'main' into receivertest-otelconsumer

9563270

Merge branch 'main' into receivertest-otelconsumer

b5d40b5

add a mutex since Generate runs in parallel

5e4520a

seems like returning an error in the output disables retries

9e47bee

go mod tidy

f30e5a8

mauri870 added this to the otel milestone Jul 30, 2025

mauri870 self-assigned this Jul 30, 2025

mauri870 added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-9 Automated backport with mergify to all the active 9.[0-9]+ branches backport-8.19 Automated backport to the 8.19 branch labels Jul 30, 2025

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Jul 30, 2025

make notice

3ce4e35

mauri870 mentioned this pull request Jul 30, 2025

otel: data race in publishing pipeline with Beats receivers #45643

Closed

cmacknz mentioned this pull request Jul 30, 2025

[beats receivers] otelconsumer returns an error on batch publish failure which disables retries #45652

Closed

mauri870 added 2 commits July 31, 2025 08:17

fix otelconsumer tests to asser no error

727b74d

Merge branch 'main' into receivertest-otelconsumer

582b6fb

mauri870 force-pushed the receivertest-otelconsumer branch from 85aad35 to 582b6fb Compare July 31, 2025 11:17

mauri870 commented Jul 31, 2025

View reviewed changes

x-pack/libbeat/outputs/otelconsumer/otelconsumer.go Outdated Show resolved Hide resolved

mauri870 changed the title ~~otel: use receivertest to test fbreceiver~~ otel: use receivertest to test fbreceiver, fix retries in otelconsumer Jul 31, 2025

mauri870 changed the title ~~otel: use receivertest to test fbreceiver, fix retries in otelconsumer~~ otel: fix retries in otelconsumer and add receivertest for fbreceiver Jul 31, 2025

mauri870 marked this pull request as ready for review July 31, 2025 11:38

mauri870 requested review from a team as code owners July 31, 2025 11:38

mauri870 requested review from leehinman and khushijain21 July 31, 2025 11:38

khushijain21 reviewed Aug 5, 2025

View reviewed changes

x-pack/filebeat/fbreceiver/receiver_test.go Show resolved Hide resolved

Merge branch 'main' into receivertest-otelconsumer

ef779a4

mauri870 force-pushed the receivertest-otelconsumer branch from d396bb7 to ef779a4 Compare August 5, 2025 12:42

go mod tidy

fd82343

mauri870 requested review from leehinman and khushijain21 August 5, 2025 13:44

add isReceiverTest property to otelconsumer

53b590d

khushijain21 reviewed Aug 5, 2025

View reviewed changes

x-pack/filebeat/fbreceiver/receiver_test.go Show resolved Hide resolved

khushijain21 reviewed Aug 5, 2025

View reviewed changes

x-pack/libbeat/outputs/otelconsumer/otelconsumer.go Outdated Show resolved Hide resolved

fix typo

df26f88

khushijain21 approved these changes Aug 5, 2025

View reviewed changes

add doc comment for TestConsumeContract

2d14745

mauri870 requested a review from khushijain21 August 5, 2025 15:02

mauri870 changed the title ~~otel: fix retries in otelconsumer and add receivertest for fbreceiver~~ otel: fix retries in otelconsumer and add receivertest test suite Aug 5, 2025

mauri870 added 2 commits August 5, 2025 12:25

rollback changes to go.mod and go.sum

956cb09

update NOTICE.txt

75243c4

leehinman approved these changes Aug 5, 2025

View reviewed changes

pierrehilbert approved these changes Aug 6, 2025

View reviewed changes

mauri870 merged commit ee17a83 into elastic:main Aug 6, 2025
206 checks passed

mergify bot mentioned this pull request Aug 6, 2025

[9.0](backport #45637) otel: fix retries in otelconsumer and add receivertest test suite #45773

Merged

6 tasks

This was referenced Aug 6, 2025

[9.1](backport #45637) otel: fix retries in otelconsumer and add receivertest test suite #45774

Merged

[8.19](backport #45637) otel: fix retries in otelconsumer and add receivertest test suite #45775

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

otel: fix retries in otelconsumer and add receivertest test suite #45637

otel: fix retries in otelconsumer and add receivertest test suite #45637

Uh oh!

mauri870 commented Jul 30, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 30, 2025

Uh oh!

mauri870 commented Jul 30, 2025

Uh oh!

Uh oh!

elasticmachine commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

khushijain21 left a comment

Uh oh!

leehinman left a comment

Uh oh!

pierrehilbert left a comment

Uh oh!

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

mergify bot commented Aug 6, 2025 •

edited

Loading

Uh oh!

Uh oh!

otel: fix retries in otelconsumer and add receivertest test suite #45637

otel: fix retries in otelconsumer and add receivertest test suite #45637

Uh oh!

Conversation

mauri870 commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Uh oh!

github-actions bot commented Jul 30, 2025

🤖 GitHub comments

Uh oh!

mauri870 commented Jul 30, 2025

Uh oh!

Uh oh!

elasticmachine commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

khushijain21 left a comment

Choose a reason for hiding this comment

Uh oh!

leehinman left a comment

Choose a reason for hiding this comment

Uh oh!

pierrehilbert left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Aug 6, 2025

Uh oh!

mergify bot commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Uh oh!

mauri870 commented Jul 30, 2025 •

edited

Loading

mergify bot commented Aug 6, 2025 •

edited

Loading