[Connector/Spanmetrics] Spanmetrics connector producing high datapoints ingestion rate to backend victoria metrics

[VijayPatil872]

Component(s)

connector/spanmetrics

What happened?

Description

Currently we are making use of spanmetrics connectors for generating metrics. we observed high volume of metrics being generated with spanmetrics connector. It causes high datapoints ingestion to Victoria metrics backend. To test it we compared with tempo metrics generator. For testing we use same data pipelines on both parallelly, and we recorded performance with same query on spanmetrics connector as well as tempo metric generator. so, with tempo metric generator with "traces_spanmetrics_calls_total" metrics it produces 6706 results series whereas spanmetrics connector with "calls_total" metrics produces 13363 results series with same timestamp. Also, if we compare the data points ingestion to Victoria metrics backend so, in this case tempo metrics generator produces 7.18K mean whereas spanmetris connector produces 53.6K mean which is quite high. this datapoint ingestion is with 14.2K mean span rate received. so, in this case spanmetrics produces 16K mean metrics point rate. So, with this, is it we are missing any configuration? or is it some internal configuration needed?

Steps to Reproduce

Expected Result

The metrics generation by spanmetrics connector should match with tempo metrics generator or it should near to what tempo metrics generator producing with same setup.

Actual Result

Span rate recieved

Metric points rate

Datapoints ingestion rate to victoria metrics

Collector version

0.104.0

Environment information

No response

OpenTelemetry Collector configuration

mode: "statefulset"
config:         
  exporters:
    otlphttp/spanmetrics:
      endpoint: 
      compression: gzip
      encoding: proto
      timeout: 30s
      tls:
        insecure: true

    otlphttp/servicegraph:
      endpoint: 
      compression: gzip
      encoding: proto
      timeout: 30s
      tls:
        insecure: true
  
  extensions:
    health_check:
      endpoint: ${env:MY_POD_IP}:***********

  connectors:
    spanmetrics:
      histogram:
        explicit:
          buckets: [10ms, 100ms, 250ms]
      aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
      metrics_flush_interval: 15s
      metrics_expiration: 5m
      dimensions_cache_size: 200 
      exemplars:
        enabled: true
        max_per_data_point: 5
      dimensions:
        - name: http.method
        - name: http.status_code
      events:
        enabled: true
        dimensions:
          - name: exception.type
      resource_metrics_key_attributes:
        - service.name
        - telemetry.sdk.language
        - telemetry.sdk.name
    servicegraph:
      latency_histogram_buckets: [100ms, 500ms, 1s, 5s, 10s]
      store:
        ttl: 2s
        max_items: 10

  processors:
    batch: {}

  receivers:
    otlp:
      protocols:
        http:
          endpoint: ${env:MY_POD_IP}:******
        grpc:
          endpoint: ${env:MY_POD_IP}:******
  service:
    extensions:
      - health_check

    pipelines:
      traces/connector-pipeline:
        exporters:
          - spanmetrics
          - servicegraph
        processors:
          - batch       
        receivers:
          - otlp    
      
      metrics/spanmetrics:
        exporters:
          - debug
          - otlphttp/spanmetrics
        processors:
          - batch
        receivers:
          - spanmetrics

      metrics/servicegraph:
        exporters:
          - debug
          - otlphttp/servicegraph
        processors:
          - batch
        receivers:
          - servicegraph        
                 
    telemetry:
      metrics:
        address: ${env:MY_POD_IP}:******

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

@VijayPatil872 You can check the calls_total label set, maybe some labels have high cardinality.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[portertech]

To add to what @Frapschen already mentioned, I suspect it's an 🍎 to 🍊 comparison due to the configuration of dimensions and resource attributes. I would also check to make sure tempo (and the metric generator) isn't dropping traces which could reduce counts etc.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

@VijayPatil872 Could you provide more details about your metrics? This will help us diagnose your problem more effectively.

[yusufcemalcelebi]

I had the same problem using the spanmetricsconnector and victoriametrics, here is my findings:

• Main problem here is that if you have multiple metric writers (opentelemetry-collectors), you should make sure that each of them are writing separate timeseries (different labelsets). This is about Single Writer Principle on prometheus
• To guarantee that, i'm using daemonset setup for my collectors and adding k8s.node.name as the metric label (adding this label might cause cardinality issues)
• But this is not enough, also you should list proper resource attributes in resource_metrics_key_attributes. This is needed because also spanmetricsconnector groups metric streams itself, and flushes these groups together.
• So for example, i'm using k8s.node.name and service.name to group these metrics together properly but if i also add k8s.pod.name (which is a resource attribute but not a dimension in metrics) this will also cause duplication of metric streams, and violate Single Writer Principle

So probably you're producing high datapoints because your metric writing flow is duplicating timeseries.