Excessive memory consumption when using ZSTD to receive compressed data

[djluck]

Describe the bug
We've seen excessive memory consumption that leads to OOMs when receiving ZSTD-compressed data.

Steps to reproduce

1. Enable zstd compression for the HTTP exporter in an upstream collector
2. Observe OOMs in your receiving collector!

You can observe excessive memory via pprof:

What version did you use?
v0.114.0

[atoulme]

Might this be something to report upstream to the zstd lib? https://github.com/klauspost/compress
Please attach the full pprof dump?

[djluck]

Unfortunately I can't seem to locate a copy of the original trace dump. Due to time commitments I'll be unlikely to recreate this issue, sorry that the above isn't as much to go on as would be ideal but hopefully others can provide dumps when/ if they run into this issue.

[ethercrow]

Hello together,

opentelemetry-collector/config/confighttp/compression.go

Line 54 in 9d67bd0

zr, err := zstd.NewReader(

- here a new zstd Reader is created for every incoming request, together with all internal buffers necessary for decompression.

Instead the library documentation recommends reusing those Reader objects:

// The Reset function can be used to initiate a new stream, which will considerably
// reduce the allocations normally caused by NewReader.

[ethercrow]

Maybe something along the lines of:

0002-zstd-reader-pool.patch