-
-
Notifications
You must be signed in to change notification settings - Fork 45
Open
Description
As a user I want to make high cardinality metrics optional so that it does not affect my prometheus instance.
We use krill in a test environment where we automatically create repositories for integration tests. This means this krill instance has a significant number of labels.
We noticed that the amount of metrics in the prometheus instance monitoring this krill instance exploded. When debugging what metrics had high cardinality, we found that the krill metrics were slowest:
Error fetching labels for krill_repo_objects: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_objects'
Error fetching labels for krill_repo_size: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_size'

Metadata
Metadata
Assignees
Labels
No labels