Skip to content

Drop data during an interval of time after an unrecoverable export error #40074

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
iblancasa opened this issue May 14, 2025 · 2 comments · Fixed by #40075
Closed

Drop data during an interval of time after an unrecoverable export error #40074

iblancasa opened this issue May 14, 2025 · 2 comments · Fixed by #40075
Assignees
Labels
enhancement New feature or request exporter/coralogix

Comments

@iblancasa
Copy link
Contributor

Component(s)

exporter/coralogix

Is your feature request related to a problem? Please describe.

We would like to improve the resilience and efficiency of the Coralogix exporter by introducing temporary data dropping behavior when specific persistent errors are encountered. The motivation for this change is to reduce unnecessary load on the Coralogix backend in scenarios where continued retries are futile or harmful, such as authentication failures or quota violations.

Describe the solution you'd like

Implement a mechanism in the Coralogix exporter that will:

Identify certain "hard failure" responses from the Coralogix backend, specifically:

  • 401 Unauthorized or 403 Forbidden (indicating authentication/authorization failure)
  • 429 Too Many Requests or other quota-related responses

Upon receiving one of these errors, suppress/delay further data exports for a fixed cooldown period, proposed to be 1 minute.

During this cooldown, all telemetry data should be dropped, not queued or retried.

This strategy aligns with best practices for exporters interacting with third-party services and helps reduce avoidable system strain.

Describe alternatives you've considered

No response

Additional context

No response

@iblancasa iblancasa self-assigned this May 14, 2025
@iblancasa iblancasa added enhancement New feature or request exporter/coralogix labels May 14, 2025
Copy link
Contributor

Pinging code owners for exporter/coralogix: @povilasv @iblancasa @douglascamata. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@mx-psi mx-psi closed this as completed in 71e89b4 May 21, 2025
dragonlord93 pushed a commit to dragonlord93/opentelemetry-collector-contrib that referenced this issue May 23, 2025
…ported by the backend (open-telemetry#40075)

#### Description
Drop telemetry data during 1 minute window when authorization or quota
errors are returned by the backend.

Merge after open-telemetry#40044.

#### Link to tracking issue
Fixes open-telemetry#40074

#### Testing
- Added new unit tests

Signed-off-by: Israel Blancas <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/coralogix
Projects
None yet
1 participant