The Prometheus receiver outputs unstable attributes when using k8s sd config, causing the cumulative to delta processor to fail to calculate delta values properly. #37413
Description
Component(s)
receiver/prometheus, processor/cumulativetodelta
What happened?
Description
Sometimes, attributes output are unstable:
- When a Pod's OwnerReference is modified, and we continue to collect metrics for that Pod at this point, some resource attributes in the output, such as
k8s.replicaset.name, will change. - Labels of a pod may change while running. If we add k8s labels by relabel in Prometheus's config, the data point attributes will be changed.
As long as some attributes are changed, the cumulative to delta processor will not calculate the delta value correctly.
Steps to Reproduce
- Start a Pod which provides metrics via Prometheus endpoint.
- Collect the metrics, and convert them to delta.
- Edit the OwnerReference of it.
3 (Alternative). Edit some k8s labels, which are added to the attributes by Prometheus relabel configuration. - Check the result of converting.
Expected Result
It should calculate the delta values correctly.
Actual Result
It cannot calculate the delta values. Because of the default behavior for start_timestamp, original values are preserved as delta.
Collector version
v0.118.0
Environment information
Environment
OS: Alpine Linux
Compiler(if manually compiled): (e.g., "go 14.2")
OpenTelemetry Collector configuration
Log output
Additional context
I think the best way is to calculate the delta values before adding any extra information. This means to make cumulative to delta processor embedded into prometheus receiver, which is not too easy.
Alternatively, I think we can add an option in cumulative to delta processor, to exclude some attributes. Then users should exclude those attributes that may change when running, to avoid this issue.