Inbound span attributes needed for SignalFx Exporter correlation to work??

[notcool11]

Looking at the sfx exporter, specifically the AddSpans function

This looks like the place where trace and metric data are correlated together. Need some guidance and/or told what I'm doing is stupid and will never work.

We have a Trace Collector (Splunk version 0.31.1) deployed in OpenShift that exposes the otlp receiver. We can send traces to that thing and using the signalfx exporter can see the traces showing up in Splunk Observability. Sweet!!

We have a separate Metrics Collector (Splunk version 0.31.1) deployed in OpenShift as a daemonset that pulls the metrics and sends to Splunk Observability as well. Also sweet!!

Now we are trying to correlate the traces with the metrics and not having a lot of luck. From looking at the code it seems like the trace/metric correlation relies on the trace having k8s.pod.id and k8s.container.id as the default correlation. But it also looks like it is comparing to a Resource list??

In our case this will never match because the Trace and Metric collectors are separate implementations. How do we proceeded?? Do we have to have the traces flow through the same collector that gathers the metrics?? Would love to keep them separate but if that is not possible we'll need to reconsider our design. Thanks.

[rmfitzpatrick]

@notcool11, thanks for your question. The host information is gathered from the available resource attributes for the batch/trace/metric data:

opentelemetry-collector-contrib/internal/splunk/hostid.go

Line 50 in abd11c1

func ResourceToHostID(res pdata.Resource) (HostID, bool) {

. As long as both metric and trace pipelines utilize the same resource/resourcedetection processors that result in the matching relevant resource attributes the correlation should apply. The Splunk distribution of the Collector has a writeup about this when using your own configuration and not the defaults that we provide: https://github.com/signalfx/splunk-otel-collector/blob/main/docs/apm-infra-correlation.md.

If you're still not seeing the desired effect would you mind providing a redacted config in a new issue in our project? You could also reach out to Splunk support at any time.

[notcool11]

Right on, thanks for the quick reply!!

Just updated our collector version and will set the yaml as the docs are recommending. Will post back with results.

[notcool11]

@rmfitzpatrick

Okay so a little good and a little confused.

Made the update to add resourcedetection to our yaml, and left the correlation stuff as default.

Now we can see Host metrics in the "Related Content" tabs on the bottom of Splunk APM and also Host Metrics on the default Service dashboard. Sweet!! Now we want more!! :)

What does it take to get other types of related content to show up?? The default correlation is on k8s.pod.uid and k8s.container.id, does that mean I have to add tags to my traces for those otel fields?? I can see my pod in IM but there is nothing linking pods to traces, would be cool to get that working as well.

Thanks!!

[rmfitzpatrick]

@notcool11 apologies for missing these updates. The feature is described at a high level here: https://docs.splunk.com/Observability/get-started/relatedcontent.html#splunk-apm. Would you be able to open a support ticket with this or further questions*, or if you prefer a new issue in our project so that we have more direct tracking: https://github.com/signalfx/splunk-otel-collector/issues?

[notcool11]

@rmfitzpatrick , I'll get that opened in the other github. I double checked our metric sets configuration and confirmed k8s.cluster.name was there and did a tag filter for the cluster. Still only seed the cluster as a related content tile. Thanks.

[notcool11]

done:
https://github.com/signalfx/splunk-otel-collector/issues/998