Skip to content

Metric explosion for krill_repo_size and krill_repo_objects metrics #1334

@ties

Description

@ties

As a user I want to make high cardinality metrics optional so that it does not affect my prometheus instance.

We use krill in a test environment where we automatically create repositories for integration tests. This means this krill instance has a significant number of labels.

We noticed that the amount of metrics in the prometheus instance monitoring this krill instance exploded. When debugging what metrics had high cardinality, we found that the krill metrics were slowest:

Error fetching labels for krill_repo_objects: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_objects'
Error fetching labels for krill_repo_size: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_size'
Image

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions