Generic proxmy configuration

[aliaksandrazherelyeu]

As part of Prometheus POC I am testing promxy to aggregate and merge metrics from two prometheus servers as shown below:

This is the config I use:

##
## Regular prometheus configuration
##
global:
  evaluation_interval: 5s
  external_labels:
    source: promxy

# Rule files specifies a list of globs. Rules and alerts are read from
# all matching files.
rule_files:
- "*rule"

# Alerting specifies settings related to the Alertmanager.
alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "127.0.0.1:12345"

# remote_write configuration is used by promxy as its local Appender, meaning all
# metrics promxy would "write" (not export) would be sent to this. Examples
# of this include: recording rules, metrics on alerting rules, etc.
remote_write:
  - url: http://localhost:8083/receive

##
### Promxy configuration
##
promxy:
  server_groups:
    # All upstream prometheus service discovery mechanisms are supported with the same
    # markup, all defined in https://github.com/prometheus/prometheus/blob/master/discovery/config/config.go#L33
    - static_configs:
        - targets:
          - 172.17.0.4:9090
          - 172.17.0.5:9090
      # labels to be added to metrics retrieved from this server_group
      labels:
        sg: localhost_9090
      # anti-affinity for merging values in timeseries between hosts in the server_group
      anti_affinity: 10s
      # time to wait for a server's response headers after fully writing the request (including its body, if any).
      # This time does not include the time to read the response body.
      timeout: 5s
      # Controls whether to use remote_read or the prom API for fetching remote RAW data (e.g. matrix selectors)
      # Note, some prometheus implementations (e.g. [VictoriaMetrics](https://github.com/prometheus/prometheus/issues/4456) don't support remote_read.
      remote_read: true
      # configures the path to send remote read requests to. The default is "api/v1/read"
      remote_read_path: api/v1/read
      # path_prefix defines a prefix to prepend to all queries to hosts in this servergroup
      # This can be relabeled using __path_prefix__
      path_prefix: /example/prefix
      # query_params adds the following map of query parameters to downstream requests.
      # The initial use-case for this is to add `nocache=1` to VictoriaMetrics downstreams
      # (see https://github.com/jacksontj/promxy/issues/202)
      query_params:
        nocache: 1
      # configures the protocol scheme used for requests. Defaults to http
      scheme: http
      # options for promxy's HTTP client when talking to hosts in server_groups
      http_client:
        # dial_timeout controls how long promxy will wait for a connection to the downstream
        # the default is 200ms.
        dial_timeout: 1s
        tls_config:
          insecure_skip_verify: true

      # relative_time_range defines a relative-to-now time range that this server group contains.
      # this is completely optional and start/end are both optional as well
      # an example is if this servergroup only has the most recent 3h of data
      # the "start" would be -3h and the end would be left out
      relative_time_range:
        start: -3h
        end: -1h
        truncate: false

      # when merging sample streams, the max value at a given timestamp will be preferred
      prefer_max: false

      # absolute_time_range defines an absolute time range that this server group contains.
      # this is completely optional and start/end are both optional as well
      # and example is if the servergroup has been deprecated and is no longer receiving data
      # you could set the specific times that it has data for.
      absolute_time_range:
        start: '2009-10-10T23:00:00Z'
        end: '2009-10-11T23:00:00Z'
        truncate: true

    # as many additional server groups as you have
    - static_configs:
        - targets:
          - localhost:9091
      labels:
        sg: localhost_9091
      anti_affinity: 10s
      scheme: http
      http_client:
        tls_config:
          insecure_skip_verify: true
      # ignore_error will make the given security group's response "optional"
      # meaning if this servergroup returns and error and others don't the overall
      # query can still succeed
      ignore_error: true

Grafana (172.17.0.3) can connect to promxy on http://172.17.0.7:8082. But when I log in promxy I don't see any targets (172.17.0.4, 172.17.0.5). In container logs I don't see any attempts of getting metrics from prometheus servers. Is there any misconfiguration?

P.S. All containers are running locally in docker desktop.

[jacksontj]

In promxy you won't see those downstreams in the targets UI (as they aren't really "scrape targets").

The config however does seem incorrect as you have a few errors:

1. you have a relative and absolute time range on the same servergroup
2. your absolute time range is from 2009 -- probably too old
3. you have skip verify TLS for HTTP endpoints

Potentially some more as well; but that is probably enough to get started