cmd/link: support for zstd-compressed DWARF sections

[stapelberg]

I recently read a blog post by @MaskRay comparing different compression algorithms for DWARF: https://maskray.me/blog/2022-09-09-zstd-compressed-debug-sections

The blog post concludes that zstd is the best choice (as it is in many other scenarios), and the author has done some work on establishing ELFCOMPRESS_ZSTD.

I don’t know if it’s too early, or if anything else is preventing us from doing so, but wanted to file this issue to track support for using zstd to make Go DWARF sections smaller and faster to write and read.

There is a good native Go implementation of zstd available: https://pkg.go.dev/github.com/klauspost/compress/zstd

[dsnet]

Would it make sense to consider a standard library "compress/zstd" package first before considering this?

[stapelberg]

Would it make sense to consider a standard library "encoding/zstd" package first before considering this?

I don’t think we need to gate landing zstd DWARF on having a zstd package in the standard library — we could just import it into internal, or vendor it, or whatever is the currently preferred approach :)

But, having encoding/zstd in the standard library would certainly be a welcome addition IMHO :)

Can you open a separate issue for that please?

[MaskRay]

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

[Foxboron]

I'd like to just point out that elfutils without support for ZSTD is going to be painfull for downstream as we can't make debug packages out of it. I'd rather have us disable compression and DWARF headers by default.

[stapelberg]

Sounds like it should be behind a flag for now, or added only once elfutils support for zstd is more widespread.

[cherrymui]

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

[dsnet]

Does the runtime ever look at the DWARF section, will we need to link in a zstd decompressor?

[cherrymui]

@dsnet the runtime doesn't look at the DWARF sections.

[mengzhuo]

@mengzhuo added ELFCOMPRESS_ZLIB support to cmd/link and may be interested:)

I'm glad to do that :) but as @dsnet suggests it looks like we need a encoding/zstd in the first place.

cc @rsc

[aarzilli]

Do the debuggers support this format? GDB? LLDB? Delve? Thanks.

cc @aarzilli @thanm

According to the blog post, it was proposed to SysV gABI in june of this year and it is supported by approximately nothing. Delve will support it when debug/elf will support it, so I think it should be added there first (or at least at the same time). Other consideration:

• I think @dsnet is right that there should be a compress/zstd first, it would be weird if the standard library supported decompressing zstd but only if it's inside an elf container
• this is all about elf compressed sections, it does not apply to zdebug sections, which are emitted for mach-o and PE, and will need to stay on zlib
• given how slow binutils moves in LTS distributions this likely can not be the default (or only) option for several years
• the list of feature requests at the end of the article is highly incomplete, for example bloaty (which the article even uses) will have to be changed to support zstd compressed elf sections
• it probably won't be a problem but compression speed is also a factor for cmd/link.