Skip to content

[exporter/elasticsearch] Optionally log documents causing indexing failures #39205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
carsonip opened this issue Apr 7, 2025 · 2 comments · Fixed by #37943
Closed

[exporter/elasticsearch] Optionally log documents causing indexing failures #39205

carsonip opened this issue Apr 7, 2025 · 2 comments · Fixed by #37943
Labels
enhancement New feature or request exporter/elasticsearch

Comments

@carsonip
Copy link
Contributor

carsonip commented Apr 7, 2025

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

Existing config telemetry::log_request_body is not very helpful in debugging because it logs all requests. It will require the user to go through MBs of logs and try to figure out the bad request causing the error.

This is ideally done before #38763 as the bump will strip out even more debugging info by default.

Describe the solution you'd like

Add an opt-in config option to log failed documents

Describe alternatives you've considered

No response

Additional context

No response
@carsonip carsonip added enhancement New feature or request needs triage New item requiring triage labels Apr 7, 2025
@carsonip
Copy link
Contributor Author

carsonip commented Apr 7, 2025

/label -needs-triage

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Apr 7, 2025

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot removed the needs triage New item requiring triage label Apr 7, 2025
@carsonip carsonip mentioned this issue Apr 7, 2025
Merged
@github-actions github-actions bot mentioned this issue Apr 8, 2025
Closed
@carsonip carsonip changed the title [exporter/elasticsearch] Log documents that led to an indexing failure [exporter/elasticsearch] Optionally log documents causing indexing failures Apr 9, 2025
andrzej-stencel pushed a commit that referenced this issue Apr 11, 2025
@carsonip
[exporter/elasticsearch] Add telemetry::log_failed_docs_input to fa…
9bb671c 
…cilitate debugging of failed bulk request items (#37943)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Add `telemetry::log_failed_docs_input` and
`telemetry::log_failed_docs_input_rate_limit` config to facilitate
debugging of failed bulk request items.
Additionally, log a warning about potentially exposing sensitive data on
any enabled `telemetry` config.

This is an improvement over using `telemetry::log_request_body` and
trying to find out the bad request causing the error out of all logged
requests.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue

Fixes
#39205

<!--Describe what testing was performed and which tests were added.-->
#### Testing

See comment

<!--Describe the documentation added.-->
#### Documentation

Updated README

<!--Please delete paragraphs that you did not use before submitting.-->
akshays-19 pushed a commit to akshays-19/opentelemetry-collector-contrib that referenced this issue Apr 23, 2025
@carsonip @AkshayS198
[exporter/elasticsearch] Add telemetry::log_failed_docs_input to fa…
5bb7a62 
…cilitate debugging of failed bulk request items (open-telemetry#37943)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Add `telemetry::log_failed_docs_input` and
`telemetry::log_failed_docs_input_rate_limit` config to facilitate
debugging of failed bulk request items.
Additionally, log a warning about potentially exposing sensitive data on
any enabled `telemetry` config.

This is an improvement over using `telemetry::log_request_body` and
trying to find out the bad request causing the error out of all logged
requests.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue

Fixes
open-telemetry#39205

<!--Describe what testing was performed and which tests were added.-->
#### Testing

See comment

<!--Describe the documentation added.-->
#### Documentation

Updated README

<!--Please delete paragraphs that you did not use before submitting.-->
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this issue Apr 24, 2025
@carsonip @Fiery-Fenix
[exporter/elasticsearch] Add telemetry::log_failed_docs_input to fa…
69bdb22 
…cilitate debugging of failed bulk request items (open-telemetry#37943)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

Add `telemetry::log_failed_docs_input` and
`telemetry::log_failed_docs_input_rate_limit` config to facilitate
debugging of failed bulk request items.
Additionally, log a warning about potentially exposing sensitive data on
any enabled `telemetry` config.

This is an improvement over using `telemetry::log_request_body` and
trying to find out the bad request causing the error out of all logged
requests.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue

Fixes
open-telemetry#39205

<!--Describe what testing was performed and which tests were added.-->
#### Testing

See comment

<!--Describe the documentation added.-->
#### Documentation

Updated README

<!--Please delete paragraphs that you did not use before submitting.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request exporter/elasticsearch
Projects
None yet
Milestone
No milestone
1 participant
@carsonip