kafkareceiver: extract Kafka message headers as client metadata #39129
Assignees
Labels
Comments
axw
added
enhancement
|
Pinging code owners:
See Adding Labels via Comments if you do not have permissions to add labels yourself. |
This was referenced Apr 3, 2025
|
Since I'm looking at the |
|
Issue filed by code owner, removing |
marclop
added a commit
to marclop/opentelemetry-collector-contrib
that referenced
this issue
Apr 8, 2025
Similar to open-telemetry#39132, but for the Kafka receiver, updates the receiver to propagate Kafka headers as client.Info (metadata). Allowing downstream processors and exporters to access the values via the enriched context. Closes open-telemetry#39129 Signed-off-by: Marc Lopez Rubio <[email protected]>
marclop
added a commit
to marclop/opentelemetry-collector-contrib
that referenced
this issue
Apr 8, 2025
Similar to open-telemetry#39132, but for the Kafka receiver, updates the receiver to propagate Kafka headers as client.Info (metadata). Allowing downstream processors and exporters to access the values via the enriched context. Closes open-telemetry#39129 Signed-off-by: Marc Lopez Rubio <[email protected]>
marclop
added a commit
to marclop/opentelemetry-collector-contrib
that referenced
this issue
Apr 8, 2025
Similar to open-telemetry#39132, but for the Kafka receiver, updates the receiver to propagate Kafka headers as client.Info (metadata). Allowing downstream processors and exporters to access the values via the enriched context. Closes open-telemetry#39129 Signed-off-by: Marc Lopez Rubio <[email protected]>
|
PR is up for review. |
LucianoGiannotti
pushed a commit
to LucianoGiannotti/opentelemetry-collector-contrib
that referenced
this issue
Apr 9, 2025
) #### Description Similar to open-telemetry#39132, but for the Kafka receiver, updates the receiver to propagate Kafka headers as client.Info (metadata). Allowing downstream processors and exporters to access the values via the enriched context. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Closes open-telemetry#39129 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests <!--Describe the documentation added.--> #### Documentation `Readme.md` to point out how it can be used with the kafka receiver. Signed-off-by: Marc Lopez Rubio <[email protected]>
atoulme
pushed a commit
that referenced
this issue
Apr 10, 2025
…cally when possible (#39145) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description The logic for pinging code owners where multiple labels matched a given component returned the first in alphabetical order. This worked generally as the first alphabetical response was the minimal matching value, as shown in #38844. However, this breaks on the kafka receiver labels. The component labels are sorted in alphabetical order by the full component name. This means `receiver/kafkametricsreceiver` is before `receiver/kafkareceiver` in the list. When the `receiver/kafka` component is referenced, the `receiver/kafkametrics` label is returned as it's the first matching label in alphabetical order. This change is to try to match component and label identically. If there's no identical match, return the first label in alphabetical order. Existing logic has been broken out into a separate script to avoid duplicating the logic. <!-- Issue number (e.g. #1234) or full URL to issue, if applicable. --> #### Link to tracking issue Unexpected behavior was found in #39129. `receiver/kafka` was the component, the label `receiver/kafkametrics` was added incorrectly, and the code owners were pinged for `receiver/kafka` correctly. When I removed the `receiver/kafkametrics` label and added `receiver/kafka`, the code owners were pinged for `receiver/kafkametrics` incorrectly. <!--Describe what testing was performed and which tests were added.--> #### Testing ``` # Testing issue where bug was detected $ .github/workflows/scripts/ping-codeowners-on-new-issue.sh No related components were given Adding the following labels: receiver/kafka Pinging code owners: - receiver/kafka: @pavolloffay @MovieStoreGuy @axw See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. # Testing other components $ export COMPONENT="os:windows" $ .github/workflows/scripts/get-label-from-component.sh $ export COMPONENT="receiver/hostmetrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/hostmetrics $ export COMPONENT="receiver/kafka" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafka $ export COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafkametrics $ export COMPONENT="extension/encoding" $ .github/workflows/scripts/get-label-from-component.sh extension/encoding $ export ISSUE=39129 $ COMPONENT="receiver/kafka" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy @axw. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. $ COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafkametrics: @dmitryax. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. ```
akshays-19
pushed a commit
to akshays-19/opentelemetry-collector-contrib
that referenced
this issue
Apr 23, 2025
…cally when possible (open-telemetry#39145) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description The logic for pinging code owners where multiple labels matched a given component returned the first in alphabetical order. This worked generally as the first alphabetical response was the minimal matching value, as shown in open-telemetry#38844. However, this breaks on the kafka receiver labels. The component labels are sorted in alphabetical order by the full component name. This means `receiver/kafkametricsreceiver` is before `receiver/kafkareceiver` in the list. When the `receiver/kafka` component is referenced, the `receiver/kafkametrics` label is returned as it's the first matching label in alphabetical order. This change is to try to match component and label identically. If there's no identical match, return the first label in alphabetical order. Existing logic has been broken out into a separate script to avoid duplicating the logic. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Unexpected behavior was found in open-telemetry#39129. `receiver/kafka` was the component, the label `receiver/kafkametrics` was added incorrectly, and the code owners were pinged for `receiver/kafka` correctly. When I removed the `receiver/kafkametrics` label and added `receiver/kafka`, the code owners were pinged for `receiver/kafkametrics` incorrectly. <!--Describe what testing was performed and which tests were added.--> #### Testing ``` # Testing issue where bug was detected $ .github/workflows/scripts/ping-codeowners-on-new-issue.sh No related components were given Adding the following labels: receiver/kafka Pinging code owners: - receiver/kafka: @pavolloffay @MovieStoreGuy @axw See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. # Testing other components $ export COMPONENT="os:windows" $ .github/workflows/scripts/get-label-from-component.sh $ export COMPONENT="receiver/hostmetrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/hostmetrics $ export COMPONENT="receiver/kafka" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafka $ export COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafkametrics $ export COMPONENT="extension/encoding" $ .github/workflows/scripts/get-label-from-component.sh extension/encoding $ export ISSUE=39129 $ COMPONENT="receiver/kafka" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy @axw. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. $ COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafkametrics: @dmitryax. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. ```
Fiery-Fenix
pushed a commit
to Fiery-Fenix/opentelemetry-collector-contrib
that referenced
this issue
Apr 24, 2025
) #### Description Similar to open-telemetry#39132, but for the Kafka receiver, updates the receiver to propagate Kafka headers as client.Info (metadata). Allowing downstream processors and exporters to access the values via the enriched context. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Closes open-telemetry#39129 <!--Describe what testing was performed and which tests were added.--> #### Testing Unit tests <!--Describe the documentation added.--> #### Documentation `Readme.md` to point out how it can be used with the kafka receiver. Signed-off-by: Marc Lopez Rubio <[email protected]>
Fiery-Fenix
pushed a commit
to Fiery-Fenix/opentelemetry-collector-contrib
that referenced
this issue
Apr 24, 2025
…cally when possible (open-telemetry#39145) <!--Ex. Fixing a bug - Describe the bug and how this fixes the issue. Ex. Adding a feature - Explain what this achieves.--> #### Description The logic for pinging code owners where multiple labels matched a given component returned the first in alphabetical order. This worked generally as the first alphabetical response was the minimal matching value, as shown in open-telemetry#38844. However, this breaks on the kafka receiver labels. The component labels are sorted in alphabetical order by the full component name. This means `receiver/kafkametricsreceiver` is before `receiver/kafkareceiver` in the list. When the `receiver/kafka` component is referenced, the `receiver/kafkametrics` label is returned as it's the first matching label in alphabetical order. This change is to try to match component and label identically. If there's no identical match, return the first label in alphabetical order. Existing logic has been broken out into a separate script to avoid duplicating the logic. <!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. --> #### Link to tracking issue Unexpected behavior was found in open-telemetry#39129. `receiver/kafka` was the component, the label `receiver/kafkametrics` was added incorrectly, and the code owners were pinged for `receiver/kafka` correctly. When I removed the `receiver/kafkametrics` label and added `receiver/kafka`, the code owners were pinged for `receiver/kafkametrics` incorrectly. <!--Describe what testing was performed and which tests were added.--> #### Testing ``` # Testing issue where bug was detected $ .github/workflows/scripts/ping-codeowners-on-new-issue.sh No related components were given Adding the following labels: receiver/kafka Pinging code owners: - receiver/kafka: @pavolloffay @MovieStoreGuy @axw See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. # Testing other components $ export COMPONENT="os:windows" $ .github/workflows/scripts/get-label-from-component.sh $ export COMPONENT="receiver/hostmetrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/hostmetrics $ export COMPONENT="receiver/kafka" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafka $ export COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/get-label-from-component.sh receiver/kafkametrics $ export COMPONENT="extension/encoding" $ .github/workflows/scripts/get-label-from-component.sh extension/encoding $ export ISSUE=39129 $ COMPONENT="receiver/kafka" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafka: @pavolloffay @MovieStoreGuy @axw. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. $ COMPONENT="receiver/kafkametrics" $ .github/workflows/scripts/ping-codeowners-issues.sh Pinging code owners for receiver/kafkametrics: @dmitryax. See [Adding Labels via Comments](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#adding-labels-via-comments) if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label. ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Component(s)
receiver/kafka
Is your feature request related to a problem? Please describe.
I would like to propagate client metadata (e.g. a tenant ID) all the way to our backend. We have Kafka in the middle of our pipeline, like this:
otlpreceiver has
include_metadataconfiguration which instructs it to record gPRC metadata (HTTP headers) as client metadata. This can be injected into out going requests using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextensionNeither kafkareceiver nor kafkaexporter currently propagate client metadata.
Describe the solution you'd like
We should add
include_metadataconfiguration to kafkareceiver to extract message headers and record them as client metadata.Describe alternatives you've considered
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: