Span Metrics Connector should initialise stale counter series at 0

[dmurphyccs]

Component(s)

connector/spanmetrics

Is your feature request related to a problem? Please describe.

The Span Metrics connector has a metrics_expiration argument which defines the time period after which metrics are considered stale and are removed from the cache.

This is a problem when using Prometheus for series that are removed frequently, because calls_total gets initialised at the value of the first scrape once the series gets registered, and Prometheus skips the jump from NaN to a given value. In order not to lose that first value, a first value of 0 should be pushed when a new series is added to the counter. Therefore, span metrics generated by the Span Metrics connector can be correctly queried in Prometheus, using PromQL functions such as increase() etc.

Describe the solution you'd like

calls_total to be initialised with a value of 0.

Describe alternatives you've considered

No response

Additional context

The same problem existed in Grafana Tempo metrics-generator but was resolved in the linked issue using the above suggested solution: grafana/tempo#2006

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[Frapschen]

make sence to me, I will try to fix it