Ability to pass the payloads though s3

[ddimitrioglo]

Background

There is an old package https://www.npmjs.com/package/sns-sqs-big-payload that supports that, but it is compatible only with AWS SDK v2

Objectives

1. Handle large payloads
2. Do not get limited by AWS's 256KB message size

[nicholasgriffintn]

Hey! Thanks for the feature suggestion.

I understand the potential use case here but I'm wondering what the advantage would be to implementing this directly into sqs-consumer vs doing this at the message handling within the client application itself / within a wrapper.

We return the message from SQS directly, so it should be possible and would keep sqs-consumer directly targeted towards its purpose, I'm not sure if we want to start doing message processing on top.

[ddimitrioglo]

Hi,

Thanks for your response.
I completely understand your concerns. That said, I’d like to offer a few additional thoughts:

• In a microservices environment, implementing support for large messages typically requires as much effort as implementing SQS polling itself. Including this logic within the library would help reduce duplication across services.

• Adding this capability would promote consistency by offering a common interface for publishing and consuming large events, which is especially valuable in teams or organizations working with shared infrastructure.

• If direct support for large message handling doesn't align with your core goals, perhaps introducing a preProcess hook or similar extension point could be a good compromise. That way, developers could plug in their own logic for handling S3-backed messages or similar use cases without altering the core flow.

I believe either of these options would provide a lot of value to the community. Appreciate you considering it!

[nicholasgriffintn]

Sorry for the delay.

Thanks for the information, this is super useful to think about.

What I need to rattle through in my brain is if this is something directly in this package or something separate from it, I wonder if some sort of wrapper around our sqs producer and consumer packages might make sense, in that way it can be purpose built for these user problems.

[nicholasgriffintn]

Also, forgot to say, there is a pre and post receive message hook already: https://bbc.github.io/sqs-consumer/interfaces/ConsumerOptions.html#postreceivemessagecallback

But also, handleMessage should already just be receiving the full message.

[nicholasgriffintn]

Here's an update, I've done a bit of thinking, there is some use to this, however, I think it will be useful for people, however, not everyone that uses our libraries.

So I do think the best option is to create a new wrapper library.

I've kicked off some initial work here: https://github.com/bbc/sqs-extended.