New component: AWS SQS Receiver

[yash025]

The purpose and use-cases of the new component

The AWS SQS receiver is designed to implement a mechanism for retrieving and processing telemetry data triggered by event notifications for S3 uploads, integrating directly with the OpenTelemetry Collector. This receiver will collect messages from Amazon Simple Queue Service (SQS) associated with S3 upload events and forward these to the OpenTelemetry Collector for further processing and exporting.

Example configuration for the component

    receivers:
      awssqsreceiver:
        queue: "https://sqs.us-west-2.amazonaws.com/xxxx/fci-raw-spans-queue-qal-usw2"
        region: "us-west-2"
        visibility_timeout: 120
        worker_pool: 2
        wait_timeout: 15

Telemetry data types supported

• traces
• metrics
• logs

Code Owner(s)

@yash025, @igrishi @MaheshGPai

Sponsor (optional)

No response

Additional context

No response

[yash025]

This has been useful for scaling our backend servers, which collect raw spans. We have one layer of lightweight collectors that collects the spans and exports them to S3 using the s3 exporter, while the second layer performs the heavy work of processing using different processors The event-driven approach has significantly helped us reduce data loss, manage retries, and better scaling.

[yash025]

#30750 (comment)

[VihasMakwana]

@yash025 #36315 seems related? Can you confirm and let me know?

[yash025]

@VihasMakwana , thanks for looking into this. Yes, it seems related, the discussion there is about updating the existing S3 receiver to listen to the SQS queue. In my opinion, updating the S3 receiver would complicate things. Since this is a common use case, does it make sense to have a separate receiver? What do you think?

[VihasMakwana]

If we add an awssqsreceiver, we shouldn't restrict it to only listening for S3 events. The receiver should be capable of handling all events from an SQS queue.

I believe it makes sense to create a new receiver specifically for reading events from SQS.

If you're willing to contribute this receiver, you'll need a sponsor. I recommend joining the OTeL Collector SIG and proposing this new component there. Additionally, it would be a good idea to reach out on Slack if possible.

[yash025]

Yes, I was thinking the same thing, to support the payload being sent directly in the message body. Let me post a message on slack.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[jamesmoessis]

Would probably make sense to have an sqs exporter as well, if you were to create a receiver.

[jamesmoessis]

@yash025 any further thoughts/movements?

[MaheshGPai]

@jamesmoessis I'm not sure if SQS exporter would be required. If I'm not wrong, SQS has a payload size limitation of 256KB. With that in place, the exporter would be inefficient. I think with the current S3 exporter, the S3 bucket can be setup to send S3 event notification and this receiver would cater to that use-case.

[jamesmoessis]

That's an interesting idea, thanks for sharing

[yash025]

@jamesmoessis I was asked for a sponsor and tried reaching out via Slack. I was asked to join a Collector SIG meeting but was unable to find the invite. Can you help me with the invite?

[jamesmoessis]

@yash025 you can find all SIG meetings listed on the community page: https://github.com/open-telemetry/community

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.