Prometheus Metrics export failed #33235
Description
Component(s)
No response
Describe the issue you're reporting
I used https://github.com/Bhoopesh123/OpenTelemetry for setting up Open telemetry on my EKS. Only prometheus metrics are failing. Im able to check the traces in jaeger.
info MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 20, "metrics": 40, "data points": 180}
2024-05-22T11:23:55.757Z error exporterhelper/common.go:95 Exporting failed. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "otlphttp/prometheus", "error": "not retryable error: Permanent error: error exporting items, request to http://opentelemetry-demo-prometheus-server:9090/api/v1/otlp/v1/metrics responded with HTTP Status Code 404", "dropped_items": 180}
go.opentelemetry.io/collector/exporter/otlphttpexporter.createMetricsExporter.WithQueue.func5.1
Prometheus server is running
banandh@MCHICL652W60 opentelemetry-app % k logs opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 -n otel-demo
ts=2024-05-21T05:33:37.140Z caller=main.go:178 level=info msg="Experimental in-memory exemplar storage enabled"
ts=2024-05-21T05:33:37.779Z caller=main.go:585 level=info msg="Starting Prometheus Server" mode=server version="(version=2.46.0, branch=HEAD, revision=cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)"
ts=2024-05-21T05:33:37.794Z caller=main.go:590 level=info build_context="(go=go1.20.6, platform=linux/arm64, user=root@42454fc0f41e, date=20230725-12:37:33, tags=netgo,builtinassets,stringlabels)"
ts=2024-05-21T05:33:37.798Z caller=main.go:591 level=info host_details="(Linux 6.6.14-0-virt Bhoopesh123/OpenTelemetry#1 SMP Fri, 26 Jan 2024 11:08:07 +0000 aarch64 opentelemetry-demo-prometheus-server-59bd64f76-w8tg6 (none))"
ts=2024-05-21T05:33:37.798Z caller=main.go:592 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2024-05-21T05:33:37.798Z caller=main.go:593 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2024-05-21T05:33:37.934Z caller=web.go:563 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2024-05-21T05:33:37.982Z caller=main.go:1026 level=info msg="Starting TSDB ..."
Im able to access prometheus in localhost:9090 after port-forward. Pls help