Zstd: Update bundled library to version 1.5.7 #18089
Open
+26,453
−7,832
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
Status: Work in Progress
alex-moch
force-pushed
the
zstd-1.5.7
branch
from
52ca857 to
03e9d2c
Compare
This commit only replaces the bundled source and does not include any ZFS integration changes. Because the build depends on integration adjustments, it will fail until the accompanying integration commit is applied. Upstream release: https://github.com/facebook/zstd/releases/tag/v1.5.7 Signed-off-by: Alexander Moch <[email protected]>
Contributor
Author
|
Memory is now unpoisoned when handing buffers out to Zstd and poisoned again when buffers are returned to the pool. Testing on the new Alpine runner looks good. I also restored SPDX license identifiers and fixed minor style issues in Compatibility with older ZFS releases and |
alex-moch
changed the title
alex-moch
force-pushed
the
zstd-1.5.7
branch
2 times, most recently
from
69e0508 to
d35d2c3
Compare
This commit builds on the previous zstd library update and adds the necessary ZFS integration and build system changes required to make zstd 1.5.7 compile and function correctly. Changes: - Add zstd_preSplit.c (new in 1.5.7) to all build systems. - Enable x86_64 assembly in userspace (huf_decompress_amd64.S). - Disable assembly in kernel for RETHUNK/IBT compatibility. - Disable intrinsics in kernel for EL10 x86_64-v3 baseline. - Disable tracing in kernel builds for AArch64 compatibility. - Fix ZSTD_isError symbol renaming with __asm__ directive. - Rename abs64 to ZSTD_abs64 (FreeBSD kernel conflict). - Fix bitstream.h attributes (MEM_STATIC -> FORCE_INLINE_TEMPLATE). - Remove xxhash.c from BSD build (now header-only). - Update symbol names in zstd_compat_wrapper.h. Kernel assembly disabled for security mitigation compatibility. User space retains full performance. Signed-off-by: Alexander Moch <[email protected]>
The Zstd context mempool can reuse buffers that were previously poisoned under AddressSanitizer, leading to false-positive use-after-poison reports during zloop and other stress tests. Explicitly unpoison memory when handing buffers out to Zstd and poison the user-visible region again when buffers are returned to the pool. This makes the allocator ASan-correct while preserving existing pooling behavior. Also fix non-standard void * pointer arithmetic in zstd_free() and remove an early return in zstd_dctx_alloc() so kmem_type/kmem_size are always set on pool hits. This only affects ASan bookkeeping in user space, does not change runtime behavior in non-ASan configurations, and does not affect on-disk formats. Signed-off-by: Alexander Moch <[email protected]>
When updating Zstandard to version 1.5.7 the SPDX license identifiers were lost. This commit restores them. Signed-off-by: Alexander Moch <[email protected]>
Dropping in `zstd-in.c` while updating Zstandard to version 1.5.7 caused checkstyle to complain about tabs vs. spaces. This commit fixes that. Signed-off-by: Alexander Moch <[email protected]>
alex-moch
force-pushed
the
zstd-1.5.7
branch
from
d35d2c3 to
492a996
Compare
Contributor
Author
|
I successfully build-tested In the latest push, tracing is disabled for kernel builds, as it caused issues on AArch64. This is now ready for review. |
github-actions
bot
added
Status: Code Review Needed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request updates the bundled Zstandard (zstd) library to version 1.5.7 and adds the necessary OpenZFS integration and build system changes required to make zstd 1.5.7 compile and function correctly.
Motivation and context
This pull request is motivated by build failures observed when building with GCC using
-Og, as reported in:The version of Zstandard currently shipped with OpenZFS ignores
-DZSTD_NO_INLINEdue to how inlining is enforced aroundxxhash.cin the upstream sources.Fixing this issue for the currently bundled Zstandard version would require carrying local patches to upstream code.
Zstandard 1.5.7, among other changes, removes the standalone
xxhash.csource and adjusts inlining behavior such that builds with-Og -g -DZSTD_NO_INLINEsucceed. As a first step, this pull request updates the bundled Zstandard version in OpenZFS to 1.5.7 and performs the required integration changes. This is considered the cleanest approach for themasterbranch.Updating the Zstandard version in released OpenZFS stable branches is likely not an option. The approach taken here is therefore to fix
masterfirst and then evaluate whether a targeted workaround is feasible for OpenZFS 2.4 and possibly earlier releases.Description
The latest upstream Zstandard 1.5.7 release has been imported into
module/zstd. Where required, corresponding changes to the OpenZFS build system and integration code have been applied.This pull request intentionally deviates from the “one commit per pull request” model:
This separation is intended to simplify review and improve traceability of upstream versus OpenZFS-specific changes.
A third commit will add SPDX license identifiers to newly imported files to match existing OpenZFS conventions.
The upstream Zstandard sources are imported without reformatting or style adjustments, which intentionally deviates from OpenZFS coding style guidelines to keep the vendor update easy to track and review.
How has this been tested?
--enable-linux-builtinTodos before marking this ready for review
Add SPDX license identifiers to newly imported files.Remove temporary allocator bypass and implement proper fix.Test against the new Alpine Linux CI runner.Perform additional compatibility checks with older ZFS releases.Check kernel builds with--enable-linux-builtin.Types of changes
Checklist
Signed-off-by.