Skip to content

Add Queuing, retry and timeout settings to AWS S3 exporter #36264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
stephenhong opened this issue Nov 7, 2024 · 11 comments
Closed

Add Queuing, retry and timeout settings to AWS S3 exporter #36264

stephenhong opened this issue Nov 7, 2024 · 11 comments
Assignees
Labels
enhancement New feature or request exporter/awss3

Comments

@stephenhong
Copy link

Component(s)

exporter/awss3

Is your feature request related to a problem? Please describe.

There is no configuration option for queuing, retry and timeout for the AWS S3 exporter. I see that the exporterhelper is part of the factory.go but when I tried to use the options in my config.yaml, the collector wouldn't start

Describe the solution you'd like

I believe adding it in the config.go like this would add the options available

Describe alternatives you've considered

No response

Additional context

If the queuing, retry and timeout settings were left out intentionally, may I ask why they had to be taken out?

@stephenhong stephenhong added enhancement New feature or request needs triage New item requiring triage labels Nov 7, 2024
Copy link
Contributor

github-actions bot commented Nov 7, 2024

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@atoulme
Copy link
Contributor

atoulme commented Nov 10, 2024

Sounds good.

@atoulme atoulme removed the needs triage New item requiring triage label Nov 10, 2024
@VihasMakwana
Copy link
Contributor

@stephenhong I can work on this, unless you're planning to implement it. Let me know!

@stephenhong
Copy link
Author

@stephenhong I can work on this, unless you're planning to implement it. Let me know!

Yes, please do. Thank you

@VihasMakwana VihasMakwana self-assigned this Nov 13, 2024
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@lopes-felipe
Copy link
Member

Queueing implemented here:
#37274

@github-actions github-actions bot removed the Stale label Jan 17, 2025
songy23 pushed a commit that referenced this issue Jan 17, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Implements the common queueing feature into the `awss3exporter`, as
suggested
[here](#36264).
chengchuanpeng pushed a commit to chengchuanpeng/opentelemetry-collector-contrib that referenced this issue Jan 26, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Implements the common queueing feature into the `awss3exporter`, as
suggested
[here](open-telemetry#36264).
@atoulme
Copy link
Contributor

atoulme commented Feb 15, 2025

Thank you. I am going to close this issue now, but if anything needs surfacing, please reach out with a new issue.

@atoulme atoulme closed this as completed Feb 15, 2025
@nimishamehta5
Copy link
Contributor

@atoulme, @pdelewski: I just came across this issue as I noticed that the timeout still isn't implemented for the s3 exporter, can that be added as well?

@atoulme
Copy link
Contributor

atoulme commented Mar 21, 2025

This was delivered as part of the PR adding sending queue support to the exporter. You can leverage the sending queue that is offered as part of this component to configure retries and timeouts. See the options of the sending queue here: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md

@nimishamehta5
Copy link
Contributor

@atoulme : the previous PR (#37274) missed implementing the timeout, and only implemented the sending queue. I've opened a PR here to add the timeout: #38873
Please take a look.

atoulme pushed a commit that referenced this issue Mar 22, 2025
#### Description

Implements the common timeout feature into the awss3exporter, as
suggested
#36264.
@namco1992
Copy link
Contributor

Hi @atoulme, I noticed that the retry settings is not mentioned anywhere, and I took a stab to clarify this and also expose the retry settings in #39509.

zeck-ops pushed a commit to zeck-ops/opentelemetry-collector-contrib that referenced this issue Apr 23, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Implements the common queueing feature into the `awss3exporter`, as
suggested
[here](open-telemetry#36264).
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this issue Apr 24, 2025
#### Description

Implements the common timeout feature into the awss3exporter, as
suggested
open-telemetry#36264.
atoulme pushed a commit that referenced this issue May 20, 2025
…koff settings (#39509)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The queue and timeout settings have been added in #36264. However, there
is no retry settings added. Initially I thought we should rely on the
standard `configretry.BackOffConfig`, and then realized that the AWS S3
client has a "standard" retryer implementation enabled by default. It
also handles the definition of retryable errors
https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry#hdr-Standard,
which makes more sense to leverage the AWS SDK instead of shoehorning it
to the standard `configretry.BackOffConfig`.

Explicitly mention the default retry mode in the documentation to avoid
confusion. Also expose the retry settings so the user can have control.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Mentioned in #36264.

<!--Describe what testing was performed and which tests were added.-->
#### Testing

<!--Describe the documentation added.-->
#### Documentation
Updated the exporter documentation to include the retry settings.

Signed-off-by: Mengnan Gong <[email protected]>
dragonlord93 pushed a commit to dragonlord93/opentelemetry-collector-contrib that referenced this issue May 23, 2025
…koff settings (open-telemetry#39509)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
The queue and timeout settings have been added in open-telemetry#36264. However, there
is no retry settings added. Initially I thought we should rely on the
standard `configretry.BackOffConfig`, and then realized that the AWS S3
client has a "standard" retryer implementation enabled by default. It
also handles the definition of retryable errors
https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry#hdr-Standard,
which makes more sense to leverage the AWS SDK instead of shoehorning it
to the standard `configretry.BackOffConfig`.

Explicitly mention the default retry mode in the documentation to avoid
confusion. Also expose the retry settings so the user can have control.

<!-- Issue number (e.g. open-telemetry#1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Mentioned in open-telemetry#36264.

<!--Describe what testing was performed and which tests were added.-->
#### Testing

<!--Describe the documentation added.-->
#### Documentation
Updated the exporter documentation to include the retry settings.

Signed-off-by: Mengnan Gong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request exporter/awss3
Projects
None yet
Development

No branches or pull requests

6 participants