Skip to content

[flaky test][receiver/github] TestEvalCommits/TestMultiplePageResponse failing intermittently #34983

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
crobert-1 opened this issue Sep 3, 2024 · 8 comments · Fixed by #37698 or #37778
Closed

[flaky test][receiver/github] TestEvalCommits/TestMultiplePageResponse failing intermittently #34983

crobert-1 opened this issue Sep 3, 2024 · 8 comments · Fixed by #37698 or #37778
Labels
flaky test a test is flaky receiver/github

Comments

@crobert-1
Copy link
Member

Component(s)

receiver/github

Describe the issue you're reporting

Failing CI/CD link

Failure output:

=== Failed
=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestMultiplePageResponse (0.00s)
    helpers_test.go:[90](https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/10682335912/job/29607989045?pr=34729#step:9:91)2: 
        	Error Trace:	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:902
make[2]: *** [../../Makefile.Common:133: test-with-cover] Error 1
        	Error:      	Not equal: 
        	            	expected: 52833113
        	            	actual  : 52833114
        	Test:       	TestEvalCommits/TestMultiplePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits (0.01s)
@crobert-1 crobert-1 added the needs triage New item requiring triage label Sep 3, 2024
@crobert-1 crobert-1 mentioned this issue Sep 3, 2024
Merged
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 3, 2024

Pinging code owners for receiver/github: @adrielp @andrzej-stencel @crobert-1. See Adding Labels via Comments if you do not have permissions to add labels yourself.

This was referenced Sep 10, 2024
Closed
Closed
This was referenced Sep 24, 2024
Closed
Closed
This was referenced Oct 8, 2024
Closed
Closed
@atoulme atoulme added flaky test a test is flaky and removed needs triage New item requiring triage labels Oct 15, 2024
@adrielp
Copy link
Contributor

adrielp commented Dec 3, 2024

I missed the ping on this, is this still an issue?

@pjanotti
Copy link
Contributor

Yes, still going on, here is a fresh one: https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/12886161926/job/35926139289#step:8:232

=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestSinglePageResponse (0.00s)
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
        	            	expected: 64928690
        	            	actual  : 64928691
make[2]: *** [../../Makefile.Common:134: test] Error 1
        	Test:       	TestEvalCommits/TestSinglePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestMultiplePageResponse (0.01s)
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
        	            	expected: 64928690
        	            	actual  : 64928691
        	Test:       	TestEvalCommits/TestMultiplePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits (0.02s)

=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestSinglePageResponse (re-run 1) (0.05s)
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
        	            	expected: 64928697
        	            	actual  : 64928698
        	Test:       	TestEvalCommits/TestSinglePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits (re-run 1) (0.06s)

@adrielp
Copy link
Contributor

adrielp commented Jan 24, 2025

Thank you @pjanotti - I'll take a look. Interesting that it's failing by 1 .

@pjanotti
Copy link
Contributor

@adrielp note that this latest failure was on Windows, my initial guess will be that the Windows CI run surfaced some concurrency issue on the test itself. This guess is based on flaky tests on Windows, I didn't look at this component/test in particular.

@adrielp
Copy link
Contributor

adrielp commented Feb 3, 2025

@pjanotti - thanks for the link & info. I'm starting to investigate. I was able to replicate it and get it to happen 2 out of 100 times on an M1 Mac.

@adrielp adrielp mentioned this issue Feb 5, 2025
Merged
andrzej-stencel pushed a commit that referenced this issue Feb 5, 2025
@adrielp
[chore] [receiver/github] fix flaky test (#37698)
8f18ec6 
#### Description

Fixes a flaky test where when with contracted resources, there was be a
one ms delay in calculation of age causing a non-equal comparison.

#### Link to tracking issue
Fixes #34983

#### Testing

Replicated tests locally. Ran 100 tests to replicate failure (2% failure
rate) and then repeated after fix (0 failures).
@pjanotti
Copy link
Contributor

pjanotti commented Feb 7, 2025

Hi @adrielp - should we reopen this one? New hit at https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/13196620198/job/36839269592#step:8:221

=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestSinglePageResponse (0.00s)
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
make[2]: *** [../../Makefile.Common:135: test] Error 1
        	            	expected: 66388426
        	            	actual  : 66388427
        	Test:       	TestEvalCommits/TestSinglePageResponse

make[1]: *** [Makefile:213: receiver/githubreceiver] Error 2
=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestMultiplePageResponse (0.01s)
make[1]: *** Waiting for unfinished jobs....
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
        	            	expected: 66388426
        	            	actual  : 66388427
        	Test:       	TestEvalCommits/TestMultiplePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits (0.01s)

=== FAIL: internal/scraper/githubscraper TestEvalCommits/TestMultiplePageResponse (re-run 1) (0.05s)
    helpers_test.go:899: 
        	Error Trace:	D:/a/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/githubreceiver/internal/scraper/githubscraper/helpers_test.go:899
        	Error:      	Not equal: 
        	            	expected: 66388443
        	            	actual  : 66388444
        	Test:       	TestEvalCommits/TestMultiplePageResponse

=== FAIL: internal/scraper/githubscraper TestEvalCommits (re-run 1) (0.05s)

@crobert-1 crobert-1 reopened this Feb 7, 2025
@adrielp adrielp mentioned this issue Feb 7, 2025
Merged
@adrielp
Copy link
Contributor

adrielp commented Feb 7, 2025

New PR: #37778

Thanks for re-opening this. I'll keep an eye open after merge. I'm surprised my local tests didn't catch it after the logic change. Though I certainly missed removing the age equal assertion.

songy23 pushed a commit that referenced this issue Feb 7, 2025
@adrielp
[chore] remove equal assertion after fixing flaky test (#37778)
6dd6bf6 
#### Description
Removes the previous assert.Equal statement that was causing the test to
intermittently fail. The equal assertion was intended to be removed
previously with the new logic but I missed it.

Will keep an eye on this because I'm surprised I couldn't get it to
happen locally (whereas before adding the new logic I could consistently
get it to fail 2% of the time.)

#### Link to tracking issue
Fixes #34983
chengchuanpeng pushed a commit to chengchuanpeng/opentelemetry-collector-contrib that referenced this issue Feb 8, 2025
@adrielp @chengchuanpeng
[chore] [receiver/github] fix flaky test (open-telemetry#37698)
c5fc01e 
#### Description

Fixes a flaky test where when with contracted resources, there was be a
one ms delay in calculation of age causing a non-equal comparison.

#### Link to tracking issue
Fixes open-telemetry#34983

#### Testing

Replicated tests locally. Ran 100 tests to replicate failure (2% failure
rate) and then repeated after fix (0 failures).
chengchuanpeng pushed a commit to chengchuanpeng/opentelemetry-collector-contrib that referenced this issue Feb 8, 2025
@adrielp @chengchuanpeng
[chore] remove equal assertion after fixing flaky test (open-telemetr…
11cda6f 
…y#37778)

#### Description
Removes the previous assert.Equal statement that was causing the test to
intermittently fail. The equal assertion was intended to be removed
previously with the new logic but I missed it.

Will keep an eye on this because I'm surprised I couldn't get it to
happen locally (whereas before adding the new logic I could consistently
get it to fail 2% of the time.)

#### Link to tracking issue
Fixes open-telemetry#34983
zeck-ops pushed a commit to zeck-ops/opentelemetry-collector-contrib that referenced this issue Apr 23, 2025
@adrielp @zeck-ops
[chore] [receiver/github] fix flaky test (open-telemetry#37698)
61b0f70 
#### Description

Fixes a flaky test where when with contracted resources, there was be a
one ms delay in calculation of age causing a non-equal comparison.

#### Link to tracking issue
Fixes open-telemetry#34983

#### Testing

Replicated tests locally. Ran 100 tests to replicate failure (2% failure
rate) and then repeated after fix (0 failures).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
flaky test a test is flaky receiver/github
Projects
None yet
Milestone
No milestone
4 participants
@atoulme @pjanotti @adrielp @crobert-1