spanmetrics prometheus counter-reset support

[povilasv]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

I have an interesting problem related to counter resets using spanmetrics with Prometheus compatible backend.

Let’s say I have an application that sends 1 span, then I restart the collector and then it will receive a span again.

In this case the metrics will look like this:


t0: calls_total 1
t1: restart
t2: calls_total 1

If you use Prometheus PromQL functions like rate / increase it will not account for any restarts. Since it only detects counter-resets if counter gets reset back to 0.

So if data would look like this:

t0: calls_total 0
t1: calls_total 1
t2: restart
t3: calls_total 0
t4: calls_total 1

Prometheus would correctly account for counter reset and increase / rate would correctly compute the value.

Then increase over time range would be equal to 2.

Similar situation happens if you enable metric_expiration and application sends spans at the interval bigger than metric_expiration.

Describe the solution you'd like

My proposal would be for spanmetrics connector to send a zero-value.
One way is when you create a new time series, you can also create a zero value at time.now() - metrics_flush_interval.

It’s a bit hacky, since you are starting time series at a past time, but in theory it is correct the previous value was “0”.

Maybe there are better proposals?

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

here is a same issue: #38537

[povilasv]

Yup looks like same PR should solve it, thanks!