Sidecar container terminates before the main container and this ends up timing out

[paolostancato]

Component(s)

collector

What happened?

Description

I have an application running in kubernetes that is instrumented with the otel sdk. I also created a sidecar OTLP collector using the operator, and annotated the deployments I want to be collected. Everything is working fine, only that every time a pod is terminated (usually due to autoscaling activity), the main container takes a couple of seconds to finish while the otc-container terminates almost immediately, which causes a timeout when the main app tries to send the last batch of metrics.

Steps to Reproduce

Deploy an application that takes a few seconds to shutdown, attach a sidecar OTLP collector and terminate the pod.

Expected Result

The otc-container terminates after the main container so this can send all the remaining metrics / spans.

In kubernetes versions newer than 1.29 the sidecar container could be deployed using the sidecar feature instead as a regular container within the pod

Actual Result

The otc-container terminates before the main container and the application times out while trying to send the last batch of metrics

Kubernetes Version

v1.32.5

Operator version

0.124.0

Collector version

0.124.1

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

Log output

Additional context

No response

[iblancasa]

Adding a crosslink because you mentioned the new sidecar mode: #3356

Although very likely there is a way to fix this with the current sidecar mode (didn't check how but maybe).

[paolostancato]

Definitely, maybe a configurable delay before shutting down would do the trick.