Add Broadway instrumentation library #92
Conversation
Library to instrument Broadway topologies, including trace linking. Broadway is a tricky one to describe in OpenTelemetry. It feels like a good one to follow Messaging semantics (like Oban), though they're doesn't plainly map to it. The approach taken here are usage of custom attributes to have a good start, while waiting for those Messaging semantics to stabilize - looks like they will change a lot anyway [in some near future][1]. One alternative is to model each GenStage as it's own destination, sort-of. Another experiement done in this patch is adding some means to propagate traces as links. Broadway doesn't give great means to do without some friction. The approach here is to add a wrapper for `Broadway.start_link/2` to inject extra configuration that will take care of this context extraction and propagation. It relies on Producer [`transformers`][2] and peek on some implementation details of official Producers (all them have some behaviour-based clients we can wrap, but not exactly public API). The good thing is this is all optional - just required if span linking is desired. [1]: open-telemetry/oteps#192 [2]: https://hexdocs.pm/broadway/1.0.3/custom-producers.html
|
For the record: while a lot of this is experimental in the sense attributes, methods of propagation etc are WIP, a similar impl is in prod. |
|
Any movement on this? Is this blocked by anything? Would love to get open telemetry tracing with broadway |
|
@andrewhr any update? |
|
@tsloughter @tielur sorry for the slow answer. I plan to revisit this implementation this week, as I do need in my current job - mostly keep up to date and give a quick test on a real app. The caveat @tsloughter is the name has been captured in the meantime. @tomtaylor has been active here, so maybe he can transfer and we could overwrite with a new implementation? I would say what we have in this PR is more complete and deals with span link / propagation so it's worth :) |
|
Great. And that'd be great if @tomtaylor agrees? |
|
Hi folks - very happy to give this library name over if there's a more complete implementation. I've not properly looked at what @andrewhr has implemented, but on the surface it looks more complete, so let's do it. What's the best way to do that? |
|
Looks like we had some good movement on this about 2 weeks ago, I'd love to see this make it to the finish line. Does anyone have steps to take to get this working. Do we need this PR merge prior to the library name transfer? |
Library to instrument Broadway topologies, including trace linking.
Broadway is a tricky one to describe in OpenTelemetry. It feels like a good one to follow Messaging semantics (like Oban), though they're doesn't plainly map to it.
The approach taken here are usage of custom attributes to have a good start, while waiting for those Messaging semantics to stabilize - looks like they will change a lot anyway in some near future.
One alternative is to model each GenStage as it's own destination, sort-of.
Another experiment done in this patch is adding some means to propagate traces as links. Broadway doesn't give great means to do without some friction. The approach here is to add a wrapper for
Broadway.start_link/2to inject extra configuration that will take care of this context extraction and propagation.It relies on Producer
transformersand peek on some implementation details of official Producers (all them have somebehaviour-based clients we can wrap, but not exactly public API). The good thing is this is all optional - just required if span linking is desired.