Skip to content

kafkaexporter: inject client metadata into Kafka message headers #39130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
axw opened this issue Apr 3, 2025 · 4 comments · Fixed by #39132
Closed

kafkaexporter: inject client metadata into Kafka message headers #39130

axw opened this issue Apr 3, 2025 · 4 comments · Fixed by #39132
Assignees
@marclop
Labels
enhancement New feature or request exporter/kafka

Comments

@axw
Copy link
Contributor

axw commented Apr 3, 2025

Component(s)

exporter/kafka

Is your feature request related to a problem? Please describe.

I would like to propagate client metadata (e.g. a tenant ID) all the way to our backend. We have Kafka in the middle of our pipeline, like this:

flowchart LR
  otlpreceiver --> kafkaexporter
  kafkaexporter --> Kafka
  Kafka --> kafkareceiver
  kafkareceiver --> otlpexporter
  otlpexporter --> backend
Loading

otlpreceiver has include_metadata configuration which instructs it to record gPRC metadata (HTTP headers) as client metadata. This can be injected into out going requests using https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextension

Neither kafkareceiver nor kafkaexporter currently propagate client metadata.

Describe the solution you'd like

Add configuration to kafkaexporter for adding message headers based on client metadata, similar to headerssetterextension.

See also #39129

Describe alternatives you've considered

No response

Additional context

No response
@axw axw added enhancement New feature or request needs triage New item requiring triage labels Apr 3, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 3, 2025

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@marclop
Copy link
Contributor

marclop commented Apr 3, 2025

I'd love to pick up this work and perhaps #39129 too since they're pretty closely related.

marclop added a commit to marclop/opentelemetry-collector-contrib that referenced this issue Apr 3, 2025
@marclop
kafkaexporter: Propagate metadata keys as headers
c5ca24b 
Adds a new config option specifying a list of metadata keys that should
be propagated as Kafka message headers.

Closes open-telemetry#39130

Signed-off-by: Marc Lopez Rubio <[email protected]>
@marclop marclop mentioned this issue Apr 3, 2025
Merged
@crobert-1
Copy link
Member

Issue filed by code owner, removing needs triage

@crobert-1 crobert-1 removed the needs triage New item requiring triage label Apr 3, 2025
@marclop
Copy link
Contributor

marclop commented Apr 7, 2025

Linked PR is now ready for review.

@axw axw mentioned this issue Apr 7, 2025
Open
@github-actions github-actions bot mentioned this issue Apr 8, 2025
Closed
dmathieu pushed a commit to dmathieu/opentelemetry-collector-contrib that referenced this issue Apr 8, 2025
@marclop @dmathieu
kafkaexporter: Propagate metadata keys as headers (open-telemetry#39132)
b58e425 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Adds a new config option specifying a list of metadata keys that should
be propagated as Kafka message headers.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes open-telemetry#39130

<!--Describe what testing was performed and which tests were added.-->
#### Testing

Unit tests added

<!--Describe the documentation added.-->
#### Documentation

Updated the exporter's README.

---------

Signed-off-by: Marc Lopez Rubio <[email protected]>
LucianoGiannotti pushed a commit to LucianoGiannotti/opentelemetry-collector-contrib that referenced this issue Apr 9, 2025
@marclop @LucianoGiannotti
kafkaexporter: Propagate metadata keys as headers (open-telemetry#39132)
8db0680 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Adds a new config option specifying a list of metadata keys that should
be propagated as Kafka message headers.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes open-telemetry#39130

<!--Describe what testing was performed and which tests were added.-->
#### Testing

Unit tests added

<!--Describe the documentation added.-->
#### Documentation

Updated the exporter's README.

---------

Signed-off-by: Marc Lopez Rubio <[email protected]>
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this issue Apr 24, 2025
@marclop @Fiery-Fenix
kafkaexporter: Propagate metadata keys as headers (open-telemetry#39132)
048f0c1 
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Adds a new config option specifying a list of metadata keys that should
be propagated as Kafka message headers.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes open-telemetry#39130

<!--Describe what testing was performed and which tests were added.-->
#### Testing

Unit tests added

<!--Describe the documentation added.-->
#### Documentation

Updated the exporter's README.

---------

Signed-off-by: Marc Lopez Rubio <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@marclop marclop

Labels
enhancement New feature or request exporter/kafka
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

kafkaexporter: Propagate metadata keys as headers marclop/opentelemetry-collector-contrib
3 participants
@axw @marclop @crobert-1