Skip to content

[correctnesstests/tracing] Flaky test: TestTracingGoldenData/opencensus-zipkin #34707

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
crobert-1 opened this issue Aug 15, 2024 · 8 comments

Comments

@crobert-1
Copy link
Member

Component(s)

No response

Describe the issue you're reporting

Failing CI/CD link

Failure output (too long to paste all of it here):

panic: test timed out after 10m0s
running tests:
	TestTracingGoldenData (10m0s)
	TestTracingGoldenData/opencensus-zipkin (9m39s)
@crobert-1
Copy link
Member Author

Potentially related: #33715

Copy link
Contributor

Pinging code owners for receiver/opencensus: @open-telemetry/collector-approvers. See Adding Labels via Comments if you do not have permissions to add labels yourself.

Copy link
Contributor

Pinging code owners for exporter/zipkin: @MovieStoreGuy @andrzej-stencel @crobert-1. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@songy23
Copy link
Member

songy23 commented Sep 3, 2024

@dmitryax
Copy link
Member

@dmitryax dmitryax added the help wanted Extra attention is needed label Nov 12, 2024
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@andrzej-stencel
Copy link
Member

Here's the relevant part of the logs from https://github.com/open-telemetry/opentelemetry-collector-contrib/actions/runs/11788131851/job/32834719070?pr=36297:

Log output
=== RUN   TestTracingGoldenData/opencensus-zipkin
2024/11/12 00:09:41 
receivers:
  opencensus:
    endpoint: "127.0.0.1:37473"
exporters:
  zipkin:
    endpoint: http://127.0.0.1:43339/api/v2/spans
    format: json
processors:
  
  batch:
    send_batch_size: 1024



extensions:

service:
  telemetry:
    metrics:
      address: 127.0.0.1:41779
  extensions:
  pipelines:
    traces:
      receivers: [opencensus]
      processors: [batch]
      exporters: [zipkin]

2024/11/12 00:09:41 Starting mock backend...
2024-11-12T00:09:41.791Z	info	[email protected]/service.go:166	Setting up own telemetry...
2024-11-12T00:09:41.791Z	warn	[email protected]/service.go:221	service::telemetry::metrics::address is being deprecated in favor of service::telemetry::metrics::readers
2024-11-12T00:09:41.791Z	info	telemetry/metrics.go:70	Serving metrics	{"address": "127.0.0.1:41779", "metrics level": "Normal"}
2024-11-12T00:09:41.791Z	info	[email protected]/service.go:238	Starting otelcol...	{"Version": "latest", "NumCPU": 4}
2024-11-12T00:09:41.791Z	info	extensions/extensions.go:39	Starting extensions...
2024-11-12T00:09:41.792Z	info	[email protected]/service.go:261	Everything is ready. Begin running and processing data.
2024/11/12 00:09:42 Starting load generator at 1024 items/sec.
2024/11/12 00:09:44  | Sent:     3,907 traces (1,953/sec) | Received:     3,907 items (1,302/sec)
2024/11/12 00:09:44 Stopped generator. Sent:     3,907 traces (1,952/sec)
2024-11-12T00:09:44.791Z	info	[email protected]/collector.go:336	Received shutdown request
2024-11-12T00:09:44.791Z	info	[email protected]/service.go:303	Starting shutdown...
2024/11/12 00:09:47  | Sent:     3,907 traces (781/sec) | Received:     3,907 items (651/sec)
2024/11/12 00:09:50  | Sent:     3,907 traces (488/sec) | Received:     3,907 items (434/sec)
2024/11/12 00:09:53  | Sent:     3,907 traces (355/sec) | Received:     3,907 items (325/sec)
2024/11/12 00:09:56  | Sent:     3,907 traces (279/sec) | Received:     3,907 items (260/sec)
2024/11/12 00:09:59  | Sent:     3,907 traces (229/sec) | Received:     3,907 items (217/sec)
2024/11/12 00:10:02  | Sent:     3,907 traces (195/sec) | Received:     3,907 items (186/sec)
2024/11/12 00:10:05  | Sent:     3,907 traces (169/sec) | Received:     3,907 items (162/sec)
2024/11/12 00:10:08  | Sent:     3,907 traces (150/sec) | Received:     3,907 items (144/sec)

... LOG OUTPUT TRUNCATED ...

2024/11/12 00:19:11  | Sent:     3,907 traces (6/sec) | Received:     3,907 items (6/sec)
2024/11/12 00:19:14  | Sent:     3,907 traces (6/sec) | Received:     3,907 items (6/sec)
2024/11/12 00:19:17  | Sent:     3,907 traces (6/sec) | Received:     3,907 items (6/sec)
panic: test timed out after 10m0s
running tests:
	TestTracingGoldenData (10m0s)
	TestTracingGoldenData/opencensus-zipkin (9m39s)

goroutine 846 [running]:
testing.(*M).startAlarm.func1()
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:2366 +0x385
created by time.goFunc
	/opt/hostedtoolcache/go/1.22.8/x64/src/time/sleep.go:177 +0x2d

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc001205520, {0x40b0926?, 0x0?}, 0x4234d78)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1750 +0x3ab
testing.runTests.func1(0xc001205520)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:2161 +0x37
testing.tRunner(0xc001205520, 0xc00107fbc0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1689 +0xfb
testing.runTests(0xc001760810, {0x6b17d00, 0x2, 0x2}, {0x10d?, 0x3?, 0x6c9c3c0?})
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:2159 +0x445
testing.(*M).Run(0xc00176c500)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:2027 +0x68b
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.DoTestMain(0xc00176c500, {0x49b1d60, 0x6c9b6a0})
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/test_bed.go:72 +0xef
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/correctnesstests/traces.TestMain(...)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/correctnesstests/traces/correctness_test.go:22
main.main()
	_testmain.go:51 +0x1a5

goroutine 23 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc001128180)
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:292 +0x9f
created by go.opencensus.io/stats/view.init.0 in goroutine 1
	/home/runner/go/pkg/mod/[email protected]/stats/view/worker.go:34 +0x8d

goroutine 24 [sync.Cond.Wait, 10 minutes]:
sync.runtime_notifyListWait(0xc0011a5390, 0x0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x0?)
	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/cond.go:70 +0x85
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc0011c02d0)
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:50 +0x99
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc0011c02d0)
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:63 +0x33
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:40 +0xcf

goroutine 25 [sync.Cond.Wait, 10 minutes]:
sync.runtime_notifyListWait(0xc0011a5510, 0x0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0x0?)
	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/cond.go:70 +0x85
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc0011c03f0)
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:50 +0x99
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc0011c03f0)
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:63 +0x33
created by github.com/cihub/seelog.NewAsyncLoopLogger in goroutine 1
	/home/runner/go/pkg/mod/github.com/cihub/[email protected]/behavior_asynclooplogger.go:40 +0xcf

goroutine 760 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7ff605bc6ea0, 0x72)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0x8?, 0xb03?, 0x0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc001e26880)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc001e26880)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc0071e4ae0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc0071e4ae0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock.go:327 +0x30
net/http.(*Server).Serve(0xc00518c1e0, {0x49b2cf0, 0xc0071e4ae0})
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3260 +0x33e
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver.(*zipkinReceiver).Start.func1()
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/zipkinreceiver/trace_receiver.go:110 +0x76
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver.(*zipkinReceiver).Start in goroutine 757
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/zipkinreceiver/trace_receiver.go:107 +0x23c

goroutine 31 [chan receive, 9 minutes]:
testing.(*T).Run(0xc0012056c0, {0xc0057414a0?, 0xc0000b6f38?}, 0xc0071e3c50)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1750 +0x3ab
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/correctnesstests/traces.TestTracingGoldenData(0xc0012056c0)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/correctnesstests/traces/correctness_test.go:41 +0xda
testing.tRunner(0xc0012056c0, 0x4234d78)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 1
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1742 +0x390

goroutine 757 [semacquire, 9 minutes]:
sync.runtime_Semacquire(0xc00562a840?)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:62 +0x25
sync.(*WaitGroup).Wait(0xc0057414d0?)
	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/waitgroup.go:116 +0x48
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.(*inProcessCollector).Stop(0xc0044fa900)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/in_process_collector.go:104 +0x4f
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.(*TestCase).StopAgent(0xc005789180)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/test_case.go:165 +0x24
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/correctnesstests/traces.testWithTracingGoldenDataset(0xc00579a340, {0x7ff604664f28, 0xc0071e39b0}, {0x49cb380, 0xc0071ee2a0}, {0x2492f60?, 0xc0?, 0x54f213?, 0x4a176f?}, {0xc001762b40, ...})
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/correctnesstests/traces/correctness_test.go:95 +0x77b
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/correctnesstests/traces.TestTracingGoldenData.func1(0xc00579a340?)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/correctnesstests/traces/correctness_test.go:42 +0x54
testing.tRunner(0xc00579a340, 0xc0071e3c50)
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1689 +0xfb
created by testing.(*T).Run in goroutine 31
	/opt/hostedtoolcache/go/1.22.8/x64/src/testing/testing.go:1742 +0x390

goroutine 641 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004784950, {0x49cb1c0, 0xc003539630})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 639
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 525 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc005413070, {0x49cb1c0, 0xc0026ac140})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 524
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 271 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0018c8100, {0x49cb1c0, 0xc002eecdc0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 268
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 774 [select]:
go.opentelemetry.io/collector/processor/batchprocessor.(*shard[...]).startLoop(0x49e8e20)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:199 +0x145
created by go.opentelemetry.io/collector/processor/batchprocessor.(*shard[...]).start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:185 +0x8d

goroutine 189 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004bcb410, {0x49cb1c0, 0xc004b7caf0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 187
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 190 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004bcb490, {0x49cb1c0, 0xc004b7cbe0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 187
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 270 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0018c80d0, {0x49cb1c0, 0xc002eecd70})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 268
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 188 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004bcb3e0, {0x49cb1c0, 0xc004b7ca50})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 187
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 765 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc00398e360, 0xc0017def28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc63807?, {0x12cd340?, 0xc00722b2c0?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 794 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc007506e30, {0x49cb1c0, 0xc00750c3c0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 791
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 508 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc003a907a0, {0x49cb1c0, 0xc003a8c140})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 507
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 771 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc00131d748, 0xc00131d728?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc00389e150?, {0x0?, 0x6d0f120?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 76 [syscall, 10 minutes]:
os/signal.signal_recv()
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sigqueue.go:152 +0x29
os/signal.loop()
	/opt/hostedtoolcache/go/1.22.8/x64/src/os/signal/signal_unix.go:23 +0x13
created by os/signal.Notify.func1.1 in goroutine 66
	/opt/hostedtoolcache/go/1.22.8/x64/src/os/signal/signal.go:151 +0x1f

goroutine 792 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc007506dd0, {0x49cb1c0, 0xc00750c320})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 791
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 509 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc003a907d0, {0x49cb1c0, 0xc003a8c190})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 507
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 777 [chan send, 9 minutes]:
go.opentelemetry.io/collector/service/internal/graph.(*Host).NotifyComponentStatusChange(...)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/graph/host.go:83
go.opentelemetry.io/collector/service/internal/status.(*reporter).componentFSM.func1(0x37b1a80?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/status/status.go:149 +0x25
go.opentelemetry.io/collector/service/internal/status.(*fsm).transition(0xc00747e5a0, 0xc007237290)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/status/status.go:40 +0x8b
go.opentelemetry.io/collector/service/internal/status.(*reporter).ReportStatus(0xc007464ca0, 0x3afa4e0?, 0xc007237290)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/status/status.go:129 +0x8f
go.opentelemetry.io/collector/service/internal/graph.(*HostWrapper).Report(0x6aebb10?, 0x3bd60e0?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/graph/graph.go:616 +0x32
go.opentelemetry.io/collector/component/componentstatus.ReportStatus(...)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/component/[email protected]/status.go:158
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver.(*ocReceiver).Start.func3()
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/opencensusreceiver/opencensus.go:162 +0x176
created by github.com/open-telemetry/opentelemetry-collector-contrib/receiver/opencensusreceiver.(*ocReceiver).Start in goroutine 761
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/receiver/opencensusreceiver/opencensus.go:159 +0xa85

goroutine 269 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0018c80a0, {0x49cb1c0, 0xc002eecd20})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 268
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 772 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc002495f40, 0xc002495f28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc00389e150?, {0x0?, 0x106d0f120?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 550 [select, 9 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 512
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:392 +0xe08

goroutine 793 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc007506e00, {0x49cb1c0, 0xc00750c370})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 791
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 773 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc000879748, 0xc001341f28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0x0?, {0x0?, 0xc00722b2c0?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 763 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7ff605bc6bb8, 0x72)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/netpoll.go:345 +0x85
internal/poll.(*pollDesc).wait(0xa?, 0x10?, 0x0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:84 +0x27
internal/poll.(*pollDesc).waitRead(...)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Accept(0xc00288db80)
	/opt/hostedtoolcache/go/1.22.8/x64/src/internal/poll/fd_unix.go:611 +0x2ac
net.(*netFD).accept(0xc00288db80)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/fd_unix.go:172 +0x29
net.(*TCPListener).accept(0xc002299540)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock_posix.go:159 +0x1e
net.(*TCPListener).Accept(0xc002299540)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/tcpsock.go:327 +0x30
net/http.(*Server).Serve(0xc00518c780, {0x49b2cf0, 0xc002299540})
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3260 +0x33e
net/http.(*Server).ListenAndServe(0xc00518c780)
	/opt/hostedtoolcache/go/1.22.8/x64/src/net/http/server.go:3189 +0x71
go.opentelemetry.io/collector/service/telemetry/internal/otelinit.InitPrometheusServer.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/telemetry/internal/otelinit/config.go:108 +0x74
created by go.opentelemetry.io/collector/service/telemetry/internal/otelinit.InitPrometheusServer in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/telemetry/internal/otelinit/config.go:106 +0x245

goroutine 551 [select, 9 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 512
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:392 +0xe08

goroutine 766 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0x6bd376, 0xc0017daf28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc63807?, {0xa42693?, 0xc00722b2c0?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 733 [select, 9 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 796
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:392 +0xe08

goroutine 761 [sync.Mutex.Lock, 9 minutes]:
sync.runtime_SemacquireMutex(0xc005095928?, 0x6c?, 0x7ff60494be40?)
	/opt/hostedtoolcache/go/1.22.8/x64/src/runtime/sema.go:77 +0x25
sync.(*Mutex).lockSlow(0xc007464ca0)
	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/mutex.go:171 +0x15d
sync.(*Mutex).Lock(...)
	/opt/hostedtoolcache/go/1.22.8/x64/src/sync/mutex.go:90
go.opentelemetry.io/collector/service/internal/status.(*reporter).ReportStatus(0xc007464ca0, 0xc00746b940, 0xc007541320)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/status/status.go:126 +0x4c
go.opentelemetry.io/collector/service/internal/graph.(*Graph).ShutdownAll(0xc00721b140, {0x49cb020, 0x6d0f120}, {0x499def0, 0xc007464ca0})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/graph/graph.go:474 +0x326
go.opentelemetry.io/collector/service.(*Service).Shutdown(0xc00722b040, {0x49cb020, 0x6d0f120})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/service.go:309 +0xe5
go.opentelemetry.io/collector/otelcol.(*Collector).shutdown(0xc0071f0b40, {0x49cb020, 0x6d0f120})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/collector.go:358 +0xc5
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0xc0071f0b40, {0x49cb020, 0x6d0f120})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/collector.go:344 +0x9b0
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.(*inProcessCollector).Start.func2()
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/in_process_collector.go:80 +0x68
created by github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.(*inProcessCollector).Start in goroutine 757
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/in_process_collector.go:78 +0x3a5

goroutine 770 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0x3738616638383037, 0xc001eb0f28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0x3738616638383037?, {0x0?, 0xc001eb0fd0?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 769 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc002494748, 0xc002494728?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0x0?, {0x0?, 0xc002494790?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 690 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004784980, {0x49cb1c0, 0xc003539680})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 639
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 640 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc004784920, {0x49cb1c0, 0xc0035395e0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 639
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 764 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0x0, 0xc00133cf28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc63807?, {0xc0052fe260?, 0xc00722b2c0?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 759 [select]:
github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.(*TestCase).logStats(0xc005789180)
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/test_case.go:303 +0xbc
created by github.com/open-telemetry/opentelemetry-collector-contrib/testbed/testbed.NewLoadGeneratorTestCase in goroutine 757
	/home/runner/work/opentelemetry-collector-contrib/opentelemetry-collector-contrib/testbed/testbed/test_case.go:126 +0x767

goroutine 767 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc001ef4340, 0xc002494f28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc00529fb60?, {0xc002494f98?, 0xc003aa6900?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 510 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc003a90800, {0x49cb1c0, 0xc003a8c1e0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 507
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 768 [chan receive, 9 minutes]:
go.opentelemetry.io/collector/exporter/internal/queue.(*sizedChannel[...]).pop(0xc004097198, 0xc002493f28?)
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/sized_channel.go:75 +0x48
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Read(0xc002493ee8?, {0xc002493fd0?, 0x11dcfaf?})
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/bounded_memory_queue.go:44 +0x45
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1()
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:39 +0xcc
created by go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start in goroutine 761
	/home/runner/go/pkg/mod/go.opentelemetry.io/collector/[email protected]/internal/queue/consumers.go:35 +0x45

goroutine 732 [select, 9 minutes]:
google.golang.org/grpc.newClientStreamWithParams.func4()
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:393 +0x8c
created by google.golang.org/grpc.newClientStreamWithParams in goroutine 796
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/stream.go:392 +0xe08

goroutine 527 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0054130d0, {0x49cb1c0, 0xc0026ac1e0})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 524
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a

goroutine 526 [select, 9 minutes]:
google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run(0xc0054130a0, {0x49cb1c0, 0xc0026ac190})
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:88 +0x115
created by google.golang.org/grpc/internal/grpcsync.NewCallbackSerializer in goroutine 524
	/home/runner/go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:52 +0x11a
exit status 2
FAIL	github.com/open-telemetry/opentelemetry-collector-contrib/testbed/correctnesstests/traces	600.048s

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants