Segment Breakdown Visualization Solved using Custom OpenTelemetry Collector Processor

[bhushan-amit]

Hi everyone,

A while ago, I opened this discussion regarding the challenge of achieving segment breakdown visualizations (similar to what New Relic offers) using OpenTelemetry in my opensource otel and grafana stack. To solve this visualisation problem, at the end, I was looking for a way to access the parent span’s name from child spans, to enable more meaningful aggregations and dashboards in Grafana via Prometheus span metrics.

I'm happy to share that we were able to solve this problem — and wanted to share both our approach and a reusable solution for anyone facing similar issues.

✅ Problem Recap

In a trace, child spans do not inherently carry their parent’s name — only a parent_span_id. While this is sufficient for trace reconstruction, it’s insufficient for metrics pipelines, where span relationships are no longer visible. The common solution of connecting using trace.id as a dimension creates high cardinality, which is impractical.

What we needed:

• A way to propagate the name of the root (or top-level parent) span to all child spans as a span attribute like TraceParentName
• This attribute could then be used in PromQL queries to filter, segment, and visualize traces meaningfully in dashboards

🔧 Our Solution

We built a custom OpenTelemetry Collector processor, which processes spans in two passes:

1. In the first pass, it identifies root spans (where parent_span_id is empty) and maps their span_id → span_name.

2. In the second pass, it walks up the span hierarchy and injects the root span's name into all descendant spans as an attribute TraceParentName.

This attribute is then picked up by the spanmetrics processor and exported to Prometheus, allowing us to filter and aggregate based on the root operation.

The code is available here:
📌 https://github.com/bhushan-amit/otel-traceprop-processor

📊 Visualization Impact

This small metadata injection allows you to:

• Build New Relic-style breakdown charts using Prometheus/Grafana
• Avoid high-cardinality issues that arise from using trace.id directly
• Segment metrics by the logical operation at the root of a trace

PromQL for Segment Breakdown, stacked area time graph grafana panel is below:

sum(rate(traces_span_metrics_duration_milliseconds_sum{TraceParentName="$parentTraceName", service_name="$serviceName"}[$__rate_interval])) by (span_name)
/
sum(rate(traces_span_metrics_duration_milliseconds_count{TraceParentName="$parentTraceName", service_name="$serviceName"}[$__rate_interval])) by (span_name)

It yields below results :

📦 Proposal for Inclusion

If this pattern proves useful to other users (and I hope it will!), I propose that this processor be considered for inclusion in the opentelemetry-collector-contrib repository as a generic trace attribute propagation processor.

I believe this fits well with the philosophy of OpenTelemetry — giving users the flexibility and power to shape telemetry pipelines according to their needs.

🙌 Final Thoughts

• This also serves as a practical example of writing custom processors in the OpenTelemetry Collector — which is a very powerful yet underutilized capability.

• OpenTelemetry is incredibly flexible, and this kind of customization is exactly what makes it a great choice for organizations looking to build observability pipelines that suit their exact needs.

I welcome thoughts, feedback, or other use cases where such an approach might be valuable!

—
Amit Bhushan
https://github.com/bhushan-amit
[email protected]