Prometheus Receiver cannot read common exemplars

[tcolgate]

Component(s)

receiver/prometheus

What happened?

Description

Steps to Reproduce

• Create an application that exposes a histogram with exemplars, via the prometheus metrics endpoint. The exemplar should include a label with (not limited to), a ., e.g. any of the common otel labels such as url.scheme
• Ensure model.NameValidationScheme = model.UTF8Validation is set, promhttp.HandlerOpts.EnableOpenMetrics= true` on the handler.
• Configure the collector to scrape prometheus (you can force the OpenMetrics scrape_protocol)

Expected Result

• The exemapler is exposed on /metrics, prometheus quotes the dotted label key, e.g.:

metric_milliseconds_bucket{n="1"} 0 # {"url.scheme"="abc"} 1.0

• The collector should scrape the application, read the exemplar, and pass this on through the pipeline

Actual Result

• the scrape fails with an error, debug shows that it fails when reading the quote

Cause

This appears to be a bug in the prometheus scrape code that is not catering for the quoted labels (raised prometheus/prometheus#15259)
Using the standard otel http instrumentation, it's hard to avoid the dotted form of hte quoted labels.

It is possible to just strings.Replace the dots in the opentelemetry-go client library's prometheus exporter, but this is not a general fix for UTF8 label keys.

Collector version

679374a

Environment information

go version go1.23.2 linux/amd64

OpenTelemetry Collector configuration

receivers:     
  prometheus:
        config:
          global:
            evaluation_interval: 1m
            scrape_interval: 1m
            scrape_protocols:
            - OpenMetricsText1.0.0
            - OpenMetricsText0.0.1
            scrape_timeout: 10s

Log output

WILL UPDATE ASAP

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

Thanks for the report. Lets follow-up on the prometheus issue.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

solved in prometheus. This will be fixed as part of a dependency update in the future

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.