Skip to content

[receiver/statsd] SIGSEGV when transport other than udp is set #22004

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
perk-sumo opened this issue May 16, 2023 · 5 comments · Fixed by #22074
Closed

[receiver/statsd] SIGSEGV when transport other than udp is set #22004

perk-sumo opened this issue May 16, 2023 · 5 comments · Fixed by #22074
Assignees
Labels
bug Something isn't working receiver/statsd statsd related issues

Comments

@perk-sumo
Copy link

Component(s)

receiver/statsd

What happened?

Description

otelcol-contrib crashes when transport other than udp is set for statsd receiver - checked with version 0.77

Steps to Reproduce

Set transport to something different than udp

receivers:
  statsd:
    transport: "tcp"
exporters:
  logging:
service:
  pipelines:
    metrics:
      exporters:
        - logging
      receivers:
        - statsd

Expected Result

I'd expect otelcol to gracefully shutdown

Actual Result

Collector crashes with following output:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x108b5ef7c]

Collector version

0.77.0

Environment information

Environment

OS: MacOS Ventura version 13.3.1 (a)

OpenTelemetry Collector configuration

receivers:
  statsd:
    transport: "tcp"
exporters:
  logging:
service:
  pipelines:
    metrics:
      exporters:
        - logging
      receivers:
        - statsd

Log output

2023-05-16T12:43:27.105+0200	info	service/telemetry.go:113	Setting up own telemetry...
2023-05-16T12:43:27.105+0200	info	service/telemetry.go:136	Serving Prometheus metrics	{"address": ":8888", "level": "Basic"}
2023-05-16T12:43:27.105+0200	info	[email protected]/exporter.go:286	Development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2023-05-16T12:43:27.106+0200	info	service/service.go:141	Starting otelcol-contrib...	{"Version": "0.77.0", "NumCPU": 10}
2023-05-16T12:43:27.107+0200	info	extensions/extensions.go:41	Starting extensions...
2023-05-16T12:43:27.107+0200	info	service/service.go:167	Starting shutdown...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x108b5ef7c]

goroutine 1 [running]:
github.com/open-telemetry/opentelemetry-collector-contrib/receiver/statsdreceiver.(*statsdReceiver).Shutdown(0x1400085e620, {0x10ae11f80, 0x1})
	github.com/open-telemetry/opentelemetry-collector-contrib/receiver/[email protected]/receiver.go:139 +0x2c
go.opentelemetry.io/collector/service/internal/graph.(*Graph).ShutdownAll(0x0?, {0x10b9da7a8, 0x140000740d0})
	go.opentelemetry.io/[email protected]/service/internal/graph/graph.go:319 +0xd4
go.opentelemetry.io/collector/service.(*Service).Shutdown(0x14000692c80, {0x10b9da7a8, 0x140000740d0})
	go.opentelemetry.io/[email protected]/service/service.go:173 +0xd0
go.opentelemetry.io/collector/otelcol.(*Collector).setupConfigurationComponents(0x140006bbbc0, {0x10b9da7a8, 0x140000740d0})
	go.opentelemetry.io/[email protected]/otelcol/collector.go:185 +0x494
go.opentelemetry.io/collector/otelcol.(*Collector).Run(0x140006bbbc0, {0x10b9da7a8, 0x140000740d0})
	go.opentelemetry.io/[email protected]/otelcol/collector.go:209 +0x3c
go.opentelemetry.io/collector/otelcol.NewCommand.func1(0x140001a7800, {0x108eb4596?, 0x2?, 0x2?})
	go.opentelemetry.io/[email protected]/otelcol/command.go:38 +0x70
github.com/spf13/cobra.(*Command).execute(0x140001a7800, {0x1400006e1c0, 0x2, 0x2})
	github.com/spf13/[email protected]/command.go:940 +0x5c8
github.com/spf13/cobra.(*Command).ExecuteC(0x140001a7800)
	github.com/spf13/[email protected]/command.go:1068 +0x35c
github.com/spf13/cobra.(*Command).Execute(0x14000930540?)
	github.com/spf13/[email protected]/command.go:992 +0x1c
main.runInteractive({{0x14000930540, 0x140009317a0, 0x14000930960, 0x14000930120, 0x14000931860}, {{0x108edd59b, 0xf}, {0x108f5ee2c, 0x1f}, {0x108eae1e6, ...}}, ...})
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:31 +0x40
main.run(...)
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main_others.go:11
main.main()
	github.com/open-telemetry/opentelemetry-collector-releases/contrib/main.go:24 +0x134

Additional context

The example config file does not work as it sets transport to custom_transport:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.77.0/receiver/statsdreceiver/testdata/config.yaml

@perk-sumo perk-sumo added bug Something isn't working needs triage New item requiring triage labels May 16, 2023
@github-actions github-actions bot added the receiver/statsd statsd related issues label May 16, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@callumj
Copy link
Contributor

callumj commented May 16, 2023

I suspect there is a bug where if the StatsD receiver fails to start the server it will still be shutdown but r.Server will be nil: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/statsdreceiver/receiver.go#L139

@callumj
Copy link
Contributor

callumj commented May 16, 2023

Can confirm, it's likely your StatsD server isn't able to actually start. When I fork statsdreceiver locally and change the following:

	if r.cancel == nil || r.server == nil {
		return nil
	}

I can now get otel collector to not panic and actually give me a reason why the listener cannot start:



[2023-05-16T15:38:09Z] Error: cannot start pipelines: listen udp 127.0.0.1:8125: bind: address already in use
--
  | [2023-05-16T15:38:09Z] 2023/05/16 15:36:10 collector server run finished with error: cannot start pipelines: listen udp 127.0.0.1:8125: bind: address already in use

@atoulme
Copy link
Contributor

atoulme commented May 16, 2023

Thanks @callumj would you like to open a PR to fix the panic?

@atoulme atoulme removed the needs triage New item requiring triage label May 16, 2023
@callumj
Copy link
Contributor

callumj commented May 16, 2023

Happy to, I’ll open one tomorrow (EDT).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working receiver/statsd statsd related issues
Projects
None yet
3 participants