[receiver/prometheusreceiver] How to add Kubestate mertrics and metric server as prometheus targets in prometheus receiver

[developer1622]

Component(s)

receiver/prometheus

Is your feature request related to a problem? Please describe.

I am unsure about this, and I do not think it is a problem; however, if it is trivial, please close this issue with a comment on how to achieve it using the K8s cluster.

So, I want to fetch metrics from Kube State metrics and the metric server but I am running into various issues if I keep the Prometheus configuration in the Prometheus receiver of Otel.

And I have one more query about k8sclusterreceiver, k8sobjectsreceiver and k8sobjectsreceiver

If I install metric-server and kube state metrics, will they overlap on metrics and work they do

I am using a k3s cluster (single node).

Thank you

Describe the solution you'd like

Please give some examples of how it can be easily used.

I would love to know how to quickly set up existing and famous metrics in a K8s cluster, such as a Kube state metric and metric server.

Describe alternatives you've considered

No response

Additional context

I have a single-node K3s cluster in one of the VMs, and I am trying to fetch all the metrics with Prometheus using Otel native and leveraging the OpenTelemetry Prometheus receiver. Thank yu.

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

And I have one more query about k8sclusterreceiver, k8sobjectsreceiver and k8sobjectsreceiver

If I install metric-server and kube state metrics, will they overlap on metrics and work they do

The k8sobjectsreceiver produces logs only, so it shouldn't overlap with metrics server or KSM.

The k8sclusterreceiver is very similar to KSM.

[developer1622]

Hi @dashpole, Thank you very much for the response.

I have one more doubt about the Prometheus configuration. Please find below my Prometheus scrape configuration, where I want to get metrics of both Kube State Metrics and Metric Server(this is protected with authentication)

      prometheus:
        config:
          scrape_configs:
            - job_name: "kube-state-metrics"
              static_configs:
                - targets: ["kube-state-metrics.kube-system.svc.cluster.local:8080"]
            - job_name: "metric-server"
              scheme: https
              scrape_interval: 20s
              scrape_timeout: 10s
              authorization:
                credentials_file: "/var/run/secrets/kubernetes.io/serviceaccount/token"
                type: Bearer
              tls_config:
                ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
                insecure_skip_verify: true
              static_configs:
                - targets: ["metrics-server.kube-system.svc.cluster.local:443"]

kube-system   kube-state-metrics                   ClusterIP   None             <none>        8080/TCP,8081/TCP                                                           15d
kube-system   metrics-server                       ClusterIP   10.108.38.174    <none>        443/TCP                                                                     15d

I can see the metrics for kube-state-metrics, but for the metric server, I am facing the below error
metric server is protected(but I can see the service account in Otel deployment(Pod) has all the permissions)

Value: 0.000000
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-05-27T06:54:39.519Z	warn	internal/transaction.go:123	Failed to scrape Prometheus endpoint	{"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1716792878497, "target_labels": "{__name__=\"up\", instance=\"metrics-server.kube-system.svc.cluster.local:443\", job=\"metric-server\"}"}
2024-05-27T06:54:40.174Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 5, "data points": 5}
2024-05-27T06:54:40.174Z	info	ResourceMetrics #0
Resource SchemaURL: 

I think it is a configuration issue; I am unsure if this is the right place to ask.

If you know the issue, could you please help me fetch the metrics of the metric server?

Thanks

[dashpole]

The metrics-server doesn't expose metrics in prometheus format. It supports the k8s metrics API for things like autoscaling and kubectl top: https://github.com/kubernetes-sigs/metrics-server.

This warning from the metrics-server repo is relevant:

Metrics Server is meant only for autoscaling purposes. For example, don't use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly.

[developer1622]

Thank you very much, @dashpole; I was off, actually, to respond.

I appreciate your quick, prompt response. I'll be sure to take note of your suggestions. We can close this. Thank you once again.