[receiver/prometheusremotewrite] Support METRIC_TYPE_UNSPECIFIED

[grizzlybite]

Component(s)

receiver/prometheusremotewrite

What happened?

Description

Hi! I'm trying to forward metrics from Prometheus to OpenTelemetry Collector using the prometheusremotewritereceiver. Prometheus is receiving metrics via remote_write from several sources (mainly vmagent and prometheus-agent), then forwarding them to Otel Collector.

flowchart TD
    vma1(vmagent-01)
    vma2(vmagent-02)
    pa1(prometheus-agent-01)
    prometheus(Prometheus)
    otg(Otel Collector Gateway)
    prwr["Prometheus Remote<br>Write Receiver"]
    vma1 -->|Remote Write| prometheus
    vma2 -->|Remote Write| prometheus
    pa1  -->|Remote Write| prometheus
    prometheus -->|Remote Write| prwr
    prwr <--> otg

Loading

However, when Prometheus sends these metrics to Otel Collector, the collector rejects them with the following error: METRIC_TYPE_UNSPECIFIED

This happens for many metrics — the rejected type is always METRIC_TYPE_UNSPECIFIED

Steps to Reproduce

Configuration

Prometheus config:

global:
  scrape_interval: 15s
  scrape_timeout: 10s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

remote_write:
  - url: "http://192.168.138.91:32411/api/v1/write"
    name: "Remote write to Otelcol gateway"
    protobuf_message: io.prometheus.write.v2.Request

Collector config (relevant part):

receivers:
  prometheusremotewrite:
    endpoint: 192.168.138.91:32411
  otlp:
    protocols:
      grpc:
        endpoint: 192.168.138.91:5555
      http:
        endpoint: 192.168.138.91:32412

Expected Result

Metrics should be successfully forwarded from Prometheus to the OpenTelemetry Collector, even if the metrics were originally written to Prometheus via remote_write (e.g., by prometheus-agent or vmagent).

Actual Result

The prometheusremotewritereceiver component rejects incoming metrics from Prometheus with METRIC_TYPE_UNSPECIFIED errors.

Example log output from Prometheus:

level=ERROR component=remote remote_name="Remote write to Otelcol gateway"
err="server returned HTTP status 400 Bad Request:
unsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"angie_slabs_pages_slots_used\""

Let me know if you’d like me to provide full debug logs or a test environment to reproduce the issue.

Collector version

0.131.1

Environment information

Environment

OS: Ubuntu 24.04
Compiler: go version go1.24.4 linux/amd64

OpenTelemetry Collector configuration

Log output

Aug 07 13:33:01 test-monit-01 prometheus[186361]: time=2025-08-07T13:33:01.437+03:00 level=ERROR source=queue_manager.go:1668 msg="non-recoverable error" component=remote remote_name="Remote write to Otelcol gateway" url=http://192.168.138.91:32411/api/v1/write failedSampleCount=3995 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 400 Bad Request: unsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_mtu_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_mtu_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_mtu_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_mtu_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_mtu_bytes\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_network_name_assign_type\"\nunsupported metric type \"METRIC_TYPE"

Aug 07 13:35:12 test-monit-01 prometheus[186361]: time=2025-08-07T13:35:12.942+03:00 level=ERROR source=queue_manager.go:1668 msg="non-recoverable error" component=remote remote_name="Remote write to Otelcol gateway" url=http://192.168.138.91:32411/api/v1/write failedSampleCount=3995 failedHistogramCount=0 failedExemplarCount=0 err="server returned HTTP status 400 Bad Request: unsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_filesystem_info\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_filesystem_info\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_filesystem_info\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_filesystem_info\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_filesystem_info\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_flush_requests_time_seconds_total\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_flush_requests_time_seconds_total\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_flush_requests_time_seconds_total\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_flush_requests_time_seconds_total\"\nunsupported metric type \"METRIC_TYPE_UNSPECIFIED\" for metric \"node_disk_flush_requests_time_seconds_total\"\nunsupported metric type \"METRIC_TYPE_UNSPECI"

Additional context

Additional context

In Prometheus, when using remote_write, metric types are not explicitly defined — they are inferred based on the metric’s structure and behavior. This might be the reason why the receiver is unable to handle them.

As far as I understand, the OpenTelemetry Collector might expect the metric type to be explicitly set, which Prometheus does not do for remote_write — especially if the metrics were ingested into Prometheus via remote_write themselves (e.g., from prometheus-agent, vmagent, etc.)

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• receiver/prometheusremotewrite: @dashpole @ArthurSens @perebaj

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[perebaj]

Hey, thanks for report that, I already get in touch with David and Sens to understand this behavior.

[dashpole]

I think the prometheusremotewrite receiver should have the same behavior as the prometheusreceiver here. It should convert all unknown-typed metrics to gauges, and should set metric.metadata["prometheus.type"]="unknown". See:

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/metricfamily.go

Line 478 in bfecfaf

metric.Metadata().PutStr(prometheus.MetricMetadataTypeKey, string(mf.metadata.Type))

and

opentelemetry-collector-contrib/receiver/prometheusreceiver/internal/util.go

Lines 105 to 106 in bfecfaf

	case model.MetricTypeGauge, model.MetricTypeUnknown:
	return pmetric.MetricTypeGauge, false

[ArthurSens]

Just for some clarity @grizzlybite, have you seen our recommendations on how to configure Prometheus? We need the Type metadata to know which kind of OTel metric we need to translate to, so sending UNSPECIFIED makes things complicated in the receiver's perspective.

We've also seen some data races with the metadata-wal-records feature flag that lead to no Type being sent, even with the feature flag set.

[ArthurSens]

About accepting UNSPECIFIED as gauges, I guess that could be a good approach, but I'm not sure what is worse: not having a metric or having it incorrect. One of OTel's goals is to build automatic visualizations that are inferred from metric types, so translating metrics to incorrect types can go sideways pretty easily 🤔

[dashpole]

We put the unknown translation into the spec, so we should probably follow it here for consistency

[grizzlybite]

@ArthurSens

Just for some clarity @grizzlybite, have you seen our recommendations on how to configure Prometheus?

Yes, I’ve read the recommendations regarding --enable-feature=metadata-wal-records, and I’ve ensured that this flag is set in the Prometheus startup options.

Here’s a snippet of how Prometheus is configured on my system:

# /etc/systemd/system/prometheus.service
[Service]
EnvironmentFile=/etc/default/prometheus
ExecStart=/usr/local/bin/prometheus $OPTS

# /etc/default/prometheus
OPTS="--config.file=/opt/prometheus/prometheus.yml
      --storage.tsdb.path=/mnt/tsdb/prometheus
      --web.listen-address=192.168.138.91:9090
      --web.external-url=http://test-monit-01.local
      --web.read-timeout=5m
      --web.max-connections=512
      --storage.tsdb.retention.time=7d
      --web.enable-remote-write-receiver
      --enable-feature=metadata-wal-records
      --log.level=info"

The issue with METRIC_TYPE_UNSPECIFIED occurs when Prometheus tries to forward metrics that were previously received via remote_write.
I connected Promlens to my Prometheus instance, and it clearly shows that all metrics which were received via remote_write are missing their type.
In cases where Prometheus scrapes its own metrics, all of them are received with a defined metric type.

[ArthurSens]

I see, thanks for the extra information!

I'm unfamiliar with vmagent, but I know Prometheus can send metadata information via remote-write, even for version 1 of the protocol. Please take a look at the docs here. Have you tried adding this block to your "leaf" Prometheus?

remote_write:
  metadata_config:
    send: true

Another option is to enable remote-write v2 from the "leaf" Prometheus to the central Prometheus (also enabling the metadata feature flag). This should make the leaf Prometheus send Type information to your central Prometheus as well

[grizzlybite]

I reviewed the documentation and tried to add this block to the prometheus-agent.

In the documentation two options are described:

Configures the sending of series metadata to remote storage:

• if the prometheus.WriteRequest message was chosen.
• When io.prometheus.write.v2.Request is used, metadata is always sent.

I tried both options:

Config 1

if the prometheus.WriteRequest message was chosen.

prometheus-agent-ex1

# cat /opt/prometheus-agent/prometheus-agent.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'balancers'
    scrape_interval: 15s
    file_sd_configs:
      - files:
          - '/opt/prometheus-agent/test/balancers.yaml'
    ...

remote_write:
  - url: "http://test-monit-01.local:9090/api/v1/write"
    name: "Remote write to Prometheus"
    queue_config:
      max_shards: 40
      capacity: 10000
      max_samples_per_send: 4000
      batch_send_deadline: "10s"
    tls_config:
      insecure_skip_verify: true
    protobuf_message: prometheus.WriteRequest
    metadata_config:
      send: true
    write_relabel_configs:
      - target_label: source
        replacement: 'prometheus_remote_write_receiver'
        action: replace
      - source_labels: ['__name__']
        regex: 'prometheus_.*|go_.*'
        action: drop

Here I added:

protobuf_message: prometheus.WriteRequest
metadata_config:
  send: true

Config 2

When io.prometheus.write.v2.Request is used, metadata is always sent.

prometheus-agent-ex2

# cat /opt/prometheus-agent/prometheus-agent.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: 'balancers'
    scrape_interval: 15s
    file_sd_configs:
      - files:
          - '/opt/prometheus-agent/test/balancers.yaml'
    ...

remote_write:
  - url: "http://test-monit-01.local:9090/api/v1/write"
    name: "Remote write to Otelcol gateway"
    queue_config:
      max_shards: 40
      capacity: 10000
      max_samples_per_send: 4000
      batch_send_deadline: "10s"
    tls_config:
      insecure_skip_verify: true
    protobuf_message: io.prometheus.write.v2.Request
    write_relabel_configs:
      - target_label: source
        replacement: 'prometheus_remote_write_receiver'
        action: replace
      - source_labels: ['__name__']
        regex: 'prometheus_.*|go_.*'
        action: drop

prometheus-agent start-up options

cat /etc/default/prometheus-agent
OPTS="--config.file=/opt/prometheus-agent/prometheus-agent.yml \
     --web.listen-address=0.0.0.0:9091 \
     --agent \
     --storage.agent.path="/mnt/tsdb/prometheus-agent" \
     --storage.agent.retention.max-time=1d \
     --enable-feature=metadata-wal-records \
     --log.level=debug"

Test procedure

• Stopped the prometheus-agent service
• Stopped the prometheus service
• Cleared the data directories for both prometheus and prometheus-agent
• Restarted both services

In both cases, i waited about 5 minutes after startup, but in PromLens the metrics still appeared as unknown.
I assume the issue lies with the remote_write itself.

---

Additional info

I also noticed another point:

When Prometheus scrapes itself, not all metrics record a type - some remain undefined (for example, standard node_exporter metrics). I suspect other exporters behave similarly, so those metrics likewise won’t be accepted by the otel-collector.

Prometheus version

/usr/local/bin/prometheus --version
prometheus, version 3.5.0 (branch: HEAD, revision: 8be3a9560fbdd18a94dedec4b747c35178177202)
  build user:       root@4451b64cb451
  build date:       2025-07-14 16:15:23
  go version:       go1.24.5
  platform:         linux/amd64
  tags:             netgo,builtinassets

[ArthurSens]

We put the unknown translation into the spec, so we should probably follow it here for consistency

Fair point, let's do it!