How to distribute incoming OTEL Collector cluster traffic to receiver on all cluster members?

[jeffexplorer]

We are deploying OTEL Collector as daemonset on Kubernetes cluster and used it as an Gateway to send metrics/traces/logs to backend storage. We are also collecting host metrics using EC2 OTEL agent and export the signals thru OTLP exporter to Gateway. Do I need to set up a haproxy in Gateway as load balancer to distribute the incoming load to OTLP receiver? There are multiple receivers are listening on different ports. Is there a way to dynamically configure proxy/load balancer based on the pipeline receiver configuration?

[jpkrohling]

Load distribution across instances can be accomplished with off-the-shelf load balancers. When using OTLP gRPC, make sure your load balancer does support this protocol. I believe NGINX does have an extension for it.

[jeffexplorer]

Thank you for confirmation. I was surprised that OTEL Collector helmchart does not provide load balancer setting. Was wondering if OTEL use different mechanism to distributed the load. Should we have a more friendly setting to distribute the load based on user receiver configuration?

[jpkrohling]

Not sure I understand. There's nothing special about load balancing for the collector compared to any other gRPC or HTTP application. When there's something special, like when you want to ensure all spans for a given trace arrive at the same collector, in the case you wish to do tail-sampling, you can use the load balancer exporter. Not sure what else we could be providing you.