Prometheus receiver stops scraping all targets when Kubernetes SD change or become unreachable

[oktocat]

Describe the bug
otel-collector running with Prometheus receiver configured to scrape Prometheus-compatible endpoints discovered via kubernetes_sd_configs stops scraping when some service discovery endpoints change or become unreachable (which is naturally happening during every deployment and subsequent rolling restart).
The receiver seems to face a deadlock somewhere in updating the SD targets group.

Steps to reproduce
otel-collector config: https://gist.githubusercontent.com/oktocat/545e12bb8286cd676ccba8318a4095ef/raw/f298a32e235b55af122e92b12ff8ffdb459f6e9c/config.yaml

To trigger the issue, it's enough to initiate a rolling restart of one of the target deployments. When this happens, the collector debug logs show the following:

{"level":"info","ts":1601986494.9710436,"caller":"service/service.go:252","msg":"Everything is ready. Begin running and processing data."}


{"level":"debug","ts":1601995775.1718767,"caller":"scrape/scrape.go:1091","msg":"Scrape failed","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_pool":"oap","target":"http://10.1.37.173:1234/","err":"Get \"http://10.1.37.173:1234/\": dial tcp 10.1.37.173:1234: connect: connection refused"}
{"level":"warn","ts":1601995775.1720421,"caller":"internal/metricsbuilder.go:106","msg":"Failed to scrape Prometheus endpoint","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_timestamp":1601995775171,"target_labels":"map[component:oap instance:10.1.37.173:1234 job:oap plane:management]"}
{"level":"debug","ts":1601995776.6160927,"caller":"scrape/scrape.go:1091","msg":"Scrape failed","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_pool":"oap","target":"http://10.1.7.143:1234/","err":"Get \"http://10.1.7.143:1234/\": dial tcp 10.1.7.143:1234: connect: connection refused"}
{"level":"warn","ts":1601995776.6162364,"caller":"internal/metricsbuilder.go:106","msg":"Failed to scrape Prometheus endpoint","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_timestamp":1601995776615,"target_labels":"map[component:oap instance:10.1.7.143:1234 job:oap plane:management]"}
{"level":"debug","ts":1601995798.0816824,"caller":"scrape/scrape.go:1091","msg":"Scrape failed","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_pool":"oap","target":"http://10.1.49.45:1234/","err":"Get \"http://10.1.49.45:1234/\": context deadline exceeded"}
{"level":"debug","ts":1601995824.7997108,"caller":"discovery/manager.go:245","msg":"Discovery receiver's channel was full so will retry the next cycle","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus"}
{"level":"debug","ts":1601995829.799763,"caller":"discovery/manager.go:245","msg":"Discovery receiver's channel was full so will retry the next cycle","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus"}


(ad infinitum)

After this all Prometheus receiver scraping stops (or at least the Prometheus exporter endpoint is not updating).

What did you expect to see?
Prometheus receiver gracefully handling some targets becoming unavailable, as well as the changes in service discovery targets.

What did you see instead?
Prometheus receiver scraping stops functioning completely.

What version did you use?
from /debug/servicez:

GitHash  c8aac9e3
BuildType release
Goversion  go1.14.7
OS  linux
Architecture amd64

What config did you use?
Config: (e.g. the yaml config file)
https://gist.githubusercontent.com/oktocat/545e12bb8286cd676ccba8318a4095ef/raw/f298a32e235b55af122e92b12ff8ffdb459f6e9c/config.yaml
Environment

Goversion go1.14.7
OS linux
Architecture amd64
Kubernetes 1.17 on EKS

Additional context
The issue exists at least in 0.2.7, 0.8.0, 0.10.0 and the latest master.

[rakyll]

We are seeing this issue as well and it's affecting all the workloads that wants to export Prometheus metrics on Kubernetes.

Having quickly reviewed the discovery and scraping packages from Prometheus, the usage of these packages seems to be as expected but I quickly noticed some possible issues in the code. For example, we possibly write to an error channel that is already closed. See https://github.com/open-telemetry/opentelemetry-collector/blob/master/receiver/prometheusreceiver/metrics_receiver.go#L70-L103. I wonder if this section needs a throughout restructuring/review again. I'm not very familiar with Prometheus' discovery manager and would appreciate some help.

cc @bogdandrutu @dinooliva

[nilebox]

@rakyll @oktocat feel free to submit a PR with a fix and I can help with reviewing it.

[nilebox]

Is this issue specific to Kubernetes Endpoint objects, as in the config example (role: endpoint), or does it also affect Pod / Service / other Kubernetes targets?

[JasonXZLiu]

@nilebox This issue is affecting only Pods and Endpoints. (I don't believe the rollout restart changes the configurations of any other Kubernetes targets).

[rakyll]

@nilebox I don't have a current fix for this issue at this moment and I'm not actively looking into it, FYI. Feel free to grab it if you have context on it.

[jmacd]

The Prometheus receiver is considered high-value by many participants in the OpenTelemetry metrics community. I suspect someone will pick this up soon, and I am happy to coordinate and discuss technical details. One thing also missing from this receiver is the Prometheus up semantic convention, see the associated spec issue: open-telemetry/opentelemetry-specification#1102

[jmacd]

@alolita ^^^

[JasonXZLiu]

@jmacd We have been working on this, but I'm still trying to figure out what exactly is causing the bug. Would love some help/guidance if you have the time.

[alolita]

@jmacd we (@JasonXZLiu @alolita) will take a look at this.

[ekarlso]

👍 for the issue, we are trying to use this but we are getting bit by the same error.

[JasonXZLiu]

For the Discovery receiver's channel was full so will retry the next cycle issue, it seems like it is related to the resource memory allocated to the OTel Collector in the Kubernetes deployment. Prometheus has similar errors which can be seen here. Essentially, the server is being overloaded.

Increasing the memory should help alleviate this problem. It may also help to limit the number/types of metrics that are being scraped in the relabel_configs and metric_relabel_configs.

[0902horn]

We are seeing the same problem too.
Could we check the healthiness of prometheus receiver in health check extension? At least, otel-collector may recover from this issue automatically when the liveness probe fails if it is deployed in k8s.
Thanks.

[ekarlso]

It doesnt really seem to be anything related to the resources allocated to the Collector at least for my cause because the Operator doesn't actually set any resource limits at all.

[liamawhite]

@0902horn we have a control loop that greps the logs for the correct message then kills the pod. A horrible hack whilst we're waiting on a fix.