Post-build package compression

[bdrewery]

Suggestion based on ideas from @mjguzik:

• Either do not compress packages in make package or use a faster compression algorithm such as gzip.
• (Re)compress all packages at the end of the bulk build into the more distribution-appropriate xz format.
• Potentially only do this on highly-used packages.
• Leaf packages can be compressed with xz in make package.

xz here means whatever PKG_EXT is.

#459 likely needs to be fixed first for this to work.

[bdrewery]

< _mjg> bdrewery: so i had a quick look and it seems lz4 provides WAY faster decompressoin time than even gzip

[mat813]

Mmm, not compressing packages at all during bulk would consume really vast amount of disk space.
Using a fast algorithm like lz4 would be a good tradeoff, and it would speed up so many ports for example everything that uses texlive...

[grahamperrin]

… not compressing packages at all during bulk would consume really vast amount of disk space. …

For ZFS use cases, does #814 improve things? #794 (comment) to visualise which datasets have compression on

[brd]

@grahamperrin See #844 for more context too

[nunotexbsd]

Instead of creating a new issue maybe asking here is the best choice since this issue is related to pkg performance.

I notice that in big packages created by poudriere it takes some time because only one thread is used by xz.
Is there any chance to speed it up by using xz -T threads?