relabel_configs replacement regex failing

[lkysow]

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 15s
        scrape_timeout: 10s
      scrape_configs:
        - job_name: consul
          metrics_path: /metrics
          consul_sd_configs:
            - server: '<address>:8500'
          relabel_configs:
            - source_labels:
                - __meta_consul_tagged_address_lan
              regex: '(.*)'
              replacement: '${1}:9102'
              target_label: '__address__'
              action: 'replace'

relabel_configs with the replacement key fails to output the matched string. I just get these errors:

Get \"http://:9102/metrics\": dial tcp :9102: connect: connection refused"

The weird thing is if I don't use the replacement key:

          relabel_configs:
            - source_labels:
                - __meta_consul_tagged_address_lan
              regex: '(.*)'
              target_label: '__address__'
              action: 'replace'

Then it correctly substitutes the address (but then uses port 80 which doesn't work for my use-case).

Additional context

Full config:

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 15s
        scrape_timeout: 10s
      scrape_configs:
        - job_name: consul
          metrics_path: /metrics
          consul_sd_configs:
            - server: '<address>:8500'
          relabel_configs:
            - source_labels:
                - __meta_consul_tagged_address_lan
              regex: '(.*)'
              replacement: '${1}:9102'
              target_label: '__address__'
              action: 'replace'
  awsecscontainermetrics:
    collection_interval: 10s
processors:
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - ecs.task.memory.utilized
          - ecs.task.memory.reserved
          - ecs.task.cpu.utilized
          - ecs.task.cpu.reserved
          - ecs.task.network.rate.rx
          - ecs.task.network.rate.tx
          - ecs.task.storage.read_bytes
          - ecs.task.storage.write_bytes
exporters:
  awsprometheusremotewrite:
    endpoint: <ep>
    aws_auth:
      region: us-east-1
      service: aps
  logging:
    loglevel: info
extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679
service:
  telemetry:
    logs:
      level: debug
  extensions: [pprof, zpages, health_check]
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [logging, awsprometheusremotewrite]
    metrics/ecs:
      receivers: [awsecscontainermetrics]
      processors: [filter]
      exporters: [logging, awsprometheusremotewrite]

[lkysow]

I've confirmed that this config works with vanilla prometheus.

[bryan-aguilar]

What version of the ADOT Collector is being used?

[bryan-aguilar]

I can't say for sure, I haven't been able to test it yet but after poking through the example docs is it possible that the single quotes are causing issues?

[lkysow]

I'm building the Dockerfile from latest: public.ecr.aws/aws-observability/aws-otel-collector:latest.

is it possible that the single quotes are causing issues?

No, I've tested this with and without single quotes. And the same config works with regular prometheus (https://github.com/lkysow/containers-couch-consul-ecs/blob/main/prometheus.tf#L159-L174 - the extra escape chars there are because it's embedded in tf).

I'm happy to jump on a zoom or something to help repro. You can DM me: https://twitter.com/lkysow

[bryan-aguilar]

Gotchya, we will take a deeper look into this and get back to you if we need additional info.

[PaurushGarg]

Prometheus configuration requires the use of additional '$' operator in Collector where the configuration is interpreted as environment variables. So, here the replacement configuration should be as follows: replacement: '$${1}:9102'. The use of additional '$' operator is required for Collector only and not for Prometheus server. This should resolve the issue.
Doc link: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/prometheusreceiver/README.md

[lkysow]

Nice find! I'll try it and report back.

[lkysow]

🎉 it works, thanks so much @PaurushGarg !

My config for posterity:

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 15s
        scrape_timeout: 10s
      scrape_configs:
        - job_name: consul
          metrics_path: /metrics
          consul_sd_configs:
          - server: '<address>:8500'
          relabel_configs:
          - source_labels:
            - __meta_consul_tagged_address_lan
            regex: '(.*)'
            replacement: '$${1}:9102'
            target_label: '__address__'
            action: 'replace'
  awsecscontainermetrics:
    collection_interval: 10s
processors:
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - ecs.task.memory.utilized
          - ecs.task.memory.reserved
          - ecs.task.cpu.utilized
          - ecs.task.cpu.reserved
          - ecs.task.network.rate.rx
          - ecs.task.network.rate.tx
          - ecs.task.storage.read_bytes
          - ecs.task.storage.write_bytes
exporters:
  awsprometheusremotewrite:
    endpoint: <ep>
    aws_auth:
      region: us-east-1
      service: aps
  logging:
    loglevel: info
extensions:
  health_check:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679
service:
  telemetry:
    logs:
      level: debug
  extensions: [pprof, zpages, health_check]
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [logging, awsprometheusremotewrite]
    metrics/ecs:
      receivers: [awsecscontainermetrics]
      processors: [filter]
      exporters: [logging, awsprometheusremotewrite]