Skip to content

Introduce new exporter helper with batching option #8122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
1 task
dmitryax opened this issue Jul 22, 2023 · 10 comments
Open
1 task

Introduce new exporter helper with batching option #8122

dmitryax opened this issue Jul 22, 2023 · 10 comments
Assignees
@dmitryax
Labels
area:exporter release:required-for-ga Must be resolved before GA release
Milestone
go.opentelemetry....

Comments

@dmitryax
Copy link
Member

dmitryax commented Jul 22, 2023 •
edited by mx-psi
Loading

This is a tracking issue for introducing the new exporter helper and migrating the existing exporters to use it.

The primary reason for introducing the new exporter helper is to move the batching to the exporter side and deprecate the batch processor as part of making the delivery pipeline reliable, as reported in #7460. More details about moving batching to the exporter helper can be found in #4646.

Shifting batching to the exporter side grants us the opportunity to leverage the exporter's data model instead of relying on OTLP. As a result, we can achieve the following benefits:

  • Ability to place failed requests back into the queue without the need for converting them back to OTLP format.
  • Enhanced control in counting queue and batch sizes using basic items (like spans, data points, or log records for OTLP) tailored to different exporters, resolving the concern raised in issue Processor: Splitting summary metrics into timeseries. opentelemetry-collector-contrib#7134.
  • Optional counting of queue and batch sizes in bytes of serialized data.

Adapting to the new exporter helper requires exporter developers to implement at least two functions:

  1. Converter: to translate pdata Metrics/Traces/Logs into a user-defined Request.
  2. Request sender: to send the user-defined Request

Design document: https://docs.google.com/document/d/1uxnn5rMHhCBLP1s8K0Pg_1mAs4gCeny8OWaYvWcuibs

(edit by @mx-psi) For essential sub-issues to mark this as complete see the 'Sub-issues' list below.

Additional sub-issues to get feature parity with the batch processor:
@dmitryax dmitryax self-assigned this Jul 22, 2023
@dmitryax dmitryax mentioned this issue Jul 22, 2023
Closed
@jmacd jmacd mentioned this issue Aug 16, 2023
Closed
This was referenced Aug 16, 2023
Merged
Merged
Closed
dmitryax added a commit that referenced this issue Aug 21, 2023
@dmitryax @codeboten
[exporterhelper] New exporter helper for custom requests (#8178)
a2242f7 
Introduce a new exporter helper that operates over client-provided
requests instead of pdata. The helper user now has to provide
`Converter` - an interface with a function implementing translation of
pdata Metrics/Traces/Logs into a user-defined `Request`. `Request` is an
interface with only one required function `Export`.

It opens a door for moving batching to the exporter, where batches will
be built from client data format, instead of pdata. The batches can be
properly sized by custom request size, which can be different from OTLP.
The same custom request sizing will be applied to the sending queue. It
will also improve the performance of the sending queue retries for
non-OTLP exporters, they don't need to translate pdata on every retry.

This is an implementation alternative to
#7874 as
suggested in
#7874 (comment)

Tracking Issue:
#8122

---------

Co-authored-by: Alex Boten <[email protected]>
dmitryax added a commit that referenced this issue Aug 21, 2023
@dmitryax
[exporterhelper] Add observability to the new exporter helper (#8244)
3dad181 
This change adds collector's internal metrics and tracing to the new
request-based exporter helpers. Only those metrics and traces are added
that are already adopted by the existing exporter helpers for backward
compatibility. The new exporter helpers can and should expose more
metrics in the future, e.g. for tracking converter errors.

Tracking Issue:
#8122
@bogdandrutu
Copy link
Member

Some feedback about the interface:

  1. [Traces|Metrics|Logs]Converter can be replaced with a simple func instead of having an interface. If you need in the future to extend capabilities of that, they can be options. Comparing interfaces with nil is problematic sometimes, see https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1, referring to https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/metrics.go#L142
  2. Can you make the old NewMetricsExporter to call into NewMetricsRequestExporter to remove duplicate code and to actually start testing the new path more intensively? Also removes lots of duplicate tests.

@dmitryax
Copy link
Member Author

Thanks for the feedback!

Can you make the old NewMetricsExporter to call into NewMetricsRequestExporter to remove duplicate code and to actually start testing the new path more intensively? Also removes lots of duplicate tests.

That's what I wanted to do after #8248 is merged 👍

This was referenced Aug 24, 2023
Open
Closed
@dmitryax dmitryax mentioned this issue Aug 31, 2023
Closed
@dmitryax dmitryax mentioned this issue Sep 13, 2023
Closed
@dmitryax dmitryax mentioned this issue Sep 25, 2023
Closed
This was referenced Oct 2, 2023
Closed
Merged
dmitryax added a commit to dmitryax/opentelemetry-collector that referenced this issue Oct 26, 2023
@dmitryax
[exporterhelper] Replace experimental converter interface with function
8c52bd5 
As proposed in open-telemetry#8122 (comment)
@dmitryax dmitryax mentioned this issue Oct 26, 2023
Merged
dmitryax added a commit to dmitryax/opentelemetry-collector that referenced this issue Oct 26, 2023
@dmitryax
[exporterhelper] Replace experimental converter interface with function
2e5b127 
As proposed in open-telemetry#8122 (comment)
@dmitryax dmitryax mentioned this issue Oct 28, 2023
Closed
@dmitryax dmitryax mentioned this issue Nov 12, 2023
Merged
dmitryax added a commit that referenced this issue Nov 17, 2023
@dmitryax
[exporterhelper] Replace experimental converter interface with functi…
1e25ed7 
…on (#8764)

As proposed in
#8122 (comment)

If we need backward conversion, we will use an optional argument to the
helper function instead of an optional interface.
@atoulme atoulme added the release:required-for-ga Must be resolved before GA release label Dec 19, 2023
This was referenced Dec 19, 2023
Closed
Open
Merged
dmitryax added a commit that referenced this issue Dec 22, 2023
@dmitryax
[chore] [exporterhelper] Items based queue sizing with bounded channel (
fe2c989 
#9164)

Introduce an option to limit the queue size by the number of items
instead of number of requests. This is preliminary step for having the
exporter helper v2 with a batcher sender placed after the queue sender.
Otherwise, it'll be hard for the users to estimate the queue size based
on the number of requests without batch processor in front of it.

This change doesn't effect the existing functionality and the items
based queue limiting cannot be utilized yet.

Updates
#8122

Alternative to
#9147
bogdandrutu pushed a commit that referenced this issue Feb 3, 2024
@dmitryax
[exporterhelper] Add WithRequestQueue option to the exporter (#8853)
bdbb5f3 
Introduce a way to enable queue in the new exporter helper with a
developer interface suggested in
#8248 (comment).

The new configuration interface for the end users provides a new
`queue_size_items` option to limit the queue by a number of spans, log
records, or metric data points. The previous way to limit the queue by
number of requests is preserved under the same field, `queue_size,`
which will later be deprecated through a longer transition process.

Tracking issue:
#8122
@dmitryax dmitryax mentioned this issue Jan 6, 2025
Open
@namco1992 namco1992 mentioned this issue Jan 9, 2025
Open
@bogdandrutu bogdandrutu mentioned this issue Jan 11, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 14, 2025
@bogdandrutu
Add capability for memory and persistent queue to block when add items (
71aae79 
#12074)

This functionality is not yet exposed to the user because there is no
config available. One option to consider is to add a "blocking" bool in
the queue config.

#### Link to tracking issue

#8122
#10368

Signed-off-by: Bogdan Drutu <[email protected]>
This was referenced Jan 18, 2025
Closed
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 21, 2025
@sfc-gh-sili
[exporter][queuebatcher] set feature gate UsePullingBasedExporterQueu…
dd1d3e6 
…eBatcher to beta (#12122)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR sets feature gate UsePullingBasedExporterQueueBatcher to beta.

<!-- Issue number if applicable -->
#### Link to tracking issue
#8122

<!--Describe what testing was performed and which tests were added.-->
#### Testing

<!--Describe the documentation added.-->
#### Documentation

<!--Please delete paragraphs that you did not use before submitting.-->
@bogdandrutu bogdandrutu mentioned this issue Jan 31, 2025
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 1, 2025
@bogdandrutu
Change Queue interface to return a callback instead of an index (#12230)
4ee2b50 
This change will allow to simplify significantly the implementation for
when queue is disabled and caller must block until the batch is
constructed and processed downstream

Updates
#8122

It is ok to have this as a breaking change since Queue is still
experimental.

Signed-off-by: Bogdan Drutu <[email protected]>
@swiatekm swiatekm mentioned this issue Feb 11, 2025
Open
@sfc-gh-sili sfc-gh-sili mentioned this issue Feb 12, 2025
Closed
@Fiery-Fenix Fiery-Fenix mentioned this issue Feb 18, 2025
Merged
@iblancasa iblancasa mentioned this issue Feb 20, 2025
Closed
@axw axw mentioned this issue Feb 25, 2025
Open
@codeboten codeboten mentioned this issue Mar 24, 2025
Closed
@sfc-gh-sili sfc-gh-sili mentioned this issue Apr 2, 2025
Open
@jmacd
Copy link
Contributor

jmacd commented May 21, 2025

Can we mark this closed @dmitryax @bogdandrutu?

@jmacd jmacd mentioned this issue May 21, 2025
Merged
mx-psi added a commit to open-telemetry/opentelemetry-collector-contrib that referenced this issue Jun 6, 2025
@jmacd @mx-psi
Update OTel-Arrow exporter to use QueueBatch settings (#40211)
d7583cc 
#### Description

Following the completion of
open-telemetry/opentelemetry-collector#8122.

#### Link to tracking issue

open-telemetry/otel-arrow#240

#### Testing

✅ 

#### Documentation

✅

---------

Co-authored-by: Pablo Baeyens <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dmitryax dmitryax

Labels
area:exporter release:required-for-ga Must be resolved before GA release
Projects
Collector: v1
Status: In Progress
Milestone
go.opentelemetry.io/exporter 1.0
Development

No branches or pull requests
9 participants
@atoulme @codeboten @axw @bogdandrutu @jmacd @dmitryax @carsonip @jamesmoessis @verejoel