k8sattribute processor requirements

[tl-eirik-albrigtsen]

Hey there, I've been trying to use the k8sattributeprocessor with various parameters on 0.63 and 0.64 for the past week with only partial success.

I can see thanos, prometheus and grafana show up with enriched k8s.pod.name labels (etc) in honeycomb (after being gathered and processed in opentelemetry-collector) and these services are all written in go with the trace configs similar to the following:

    - name: THANOS_TRACING_CONFIG
      value: |
        type: JAEGER
        config:
            service_name: "storegateway"
            sampler_type: "const"
            sampler_param: 1
            endpoint: "http://otel-collector-collector.opentelemetry-operator-system.svc.cluster.local:14268/api/traces"

on the other hand, none of the microservices we have (in both dotnet and rust) seem to get the k8s attributes set even though they should be using the same trace setup (jaeger/const sampler/endpoint, all using service.name). so am wondering if there are any requirements k8sattribute processor has that I am missing? is there other attributes that go typically sends by default that i need to send?

setup

running the collector via the honeycomb/opentelemetry-collector chart with image.tag set to the latest 0.64.0. This chart does not support any daemonset shenanigans, but am guessing without passthrough and it partially working already for some services, this should not be relevant?

config:

receivers:
  jaeger:
    protocols:
      grpc: null
      thrift_binary: null
      thrift_compact: null
      thrift_http:
        endpoint: 0.0.0.0:14268
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  zipkin: {}

processors:
  batch: {}
  k8sattributes:
    auth_type: serviceAccount
    extract:
      metadata:
      - k8s.pod.name
      - k8s.deployment.name
      - k8s.node.name
      - k8s.namespace.name
    passthrough: false
  memory_limiter:
    check_interval: 5s
    limit_percentage: 75
    spike_limit_percentage: 25

exporters:
  otlp:
    endpoint: http://refinery.opentelemetry-operator-system.svc.cluster.local:8080

extensions:
  health_check:

  memory_ballast:
    size_in_percentage: 35

service:
  extensions:
  - health_check
  pipelines:
    traces:
      exporters:
      - otlphttp
      processors:
      - memory_limiter
      - batch
      - k8sattributes
      receivers:
      - otlp
      - zipkin
      - jaeger

tried many different config setups, but not found any permutations that gives me any improvement. maybe there is some magic with pod_association, but it's been kind of hard to grok from the docs