Skip to content

Add LIB_SYS_TRY_SHARED ENV var to try to link with shared library #206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

hodrob84
Copy link

@hodrob84 hodrob84 commented Aug 15, 2024

On cross-compilation libz-sys is always building static zlib library, except for Apple platforms. If target has an installed zlib shared library, this is problem, because:

  1. the binary is increased, because we are including a static zlib as well

  2. it may cause build issues if the pkg-config sets the linker arguments, like this:

    /tmp/rustc0ZqD0F/liblibz_sys-84983a050a121d20.rlib(inflate.o): relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol '__stack_chk_guard' which may bind externally can not be used when making a shared object; recompile with -fPIC

To workaround the issue, add a LIB_SYS_TRY_SHARED=1 ENV var (similarly to LIBZ_SYS_STATIC=1) to force the use of the shared library, recognized by the pkg-config.

fixes #201

On cross-compilation libz-sys is always building static zlib library,
except for Apple platforms. If target has an installed zlib shared
library, this is problem, because:

  1. the binary is increased, because we are including a static zlib as
     well
  2. it may cause build issues if the pkg-config sets the linker
     arguments, like this:

     /tmp/rustc0ZqD0F/liblibz_sys-84983a050a121d20.rlib(inflate.o): relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol '__stack_chk_guard' which may bind externally can not be used when making a shared object; recompile with -fPIC

To workaround the issue, add a LIB_SYS_TRY_SHARED=1 ENV var (similarly
to LIBZ_SYS_STATIC=1) to force the use of the shared library, recognized
by the pkg-config.

fixes rust-lang#201
@hodrob84 hodrob84 force-pushed the hodrob/try_link_with_shared_on_cc branch from 612984b to 2422982 Compare August 15, 2024 15:24
@Byron
Copy link
Member

Byron commented Aug 16, 2024

Thanks a lot for tackling this.

@jongiddy I feel that libz-sys gets more complex by adding patch upon patch, and hoped you could involve yourself here more as well. Maybe you could help with making a case for using environment variables to alter build results, or for using Cargo Features. This is my related comment, to allow for backward compatible fixes for those who know about it.

Thanks everyone for your help.

@Byron Byron added the question There is a question to be answered label Aug 16, 2024
@jongiddy
Copy link

Neither option is ideal.

Using an environment variable means there is a hidden dependency that affects the build. When anyone needs support, it is an extra relevant piece of information they need to supply. And it is easy to not be aware whether it is set or not.

Using a feature is painful when a transitive dependency is deeply nested, and libz-sys is typically deeply nested. You either need to pass the feature setting down through the intermediate dependencies or include the dependency in the top-level crate, even though it is not used directly.

Pragmatically, since there is already an environment variable for controlling static vs dynamic, I'd accept using an environment variable here. However, I suggest that the existing variable is reused, so that setting LIBZ_SYS_STATIC=0 always forces dynamic linking, even for the supposedly non-supported targets (e.g. target.contains("msvc")).

@Byron Byron removed the question There is a question to be answered label Aug 17, 2024
@wezm
Copy link
Member

wezm commented Sep 3, 2024

FWIW there is precedent in other -sys crates for disabling building of vendored C code/forcing use of system library through an env var:

  • libgit2-sys: "LIBGIT2_NO_VENDOR": "1",
  • libssh2-sys: "LIBSSH2_SYS_USE_PKG_CONFIG": "1",
  • openssl-sys: "OPENSSL_NO_VENDOR": "1",
  • pcre2-sys: "PCRE2_SYS_STATIC": "0",
  • rustonig-sys: "RUSTONIG_SYSTEM_LIBONIG": "1",
  • zstd-sys: "ZSTD_SYS_USE_PKG_CONFIG": "1",
  • libsqlite3-sys: "LIBSQLITE3_SYS_USE_PKG_CONFIG": "1",

@Byron
Copy link
Member

Byron commented Sep 3, 2024

Thanks a lot for the summary, it's good to know the standards.

This PR is at a stage where the question remains if LIBZ_SYS_STATIC=0 can be used instead of introducing a new environment variable. On top of that, I think there should be a section in the crate-level docs to document this behaviour.

@jongiddy
Copy link

jongiddy commented Sep 3, 2024

  • pcre2-sys: "PCRE2_SYS_STATIC": "0",

This one is interesting. Matches the proposed one variable model with 0 meaning force dynamic and 1 meaning force static.

@tsjk
Copy link

tsjk commented Jan 16, 2025

I'm hitting this when trying to cross-compile a project.
I have zlib1g-dev:arm64 installed on the host, but compilation fails because the compiler is called in a bad way - probably because it tries to statically link against a foreign architecture library (I see -lz).
The zlib-dependency is deep, so it´s not in the project's Cargo.toml.
Is there some way to work this around?

@Byron
Copy link
Member

Byron commented Jan 17, 2025

I think this patch is stuck on this question along with the implementation lacking discoverability as there are no docs at all.
Maybe the best discoverability could be achieved by putting typical compiler errors into the library docs, along with how to use certain environment variables to fix it.

@tsjk
Copy link

tsjk commented Jan 17, 2025

I think this patch is stuck on this question along with the implementation lacking discoverability as there are no docs at all. Maybe the best discoverability could be achieved by putting typical compiler errors into the library docs, along with how to use certain environment variables to fix it.

Is there something I can do in the meanwhile to be able to cross-compile the project I'm looking at?

@Byron
Copy link
Member

Byron commented Jan 18, 2025

One could tell Cargo to use the patched version of the dependency (with [patch."crates-io"]) to make the environment variable override available.
Alternatively, one could pick up this PR and push it to completion.

@tsjk
Copy link

tsjk commented Jan 18, 2025

One could tell Cargo to use the patched version of the dependency (with [patch."crates-io"]) to make the environment variable override available. Alternatively, one could pick up this PR and push it to completion.

In my case this does not seem to help.
I did

ENV target_host=aarch64-linux-gnu \
    target_host_rust=aarch64-unknown-linux-gnu \
    AR=${target_host}-ar \
    AS=${target_host}-as \
    CC=${target_host}-gcc \
    CXX=${target_host}-g++ \
    LD=${target_host}-ld \
    RANLIB="${target_host}-ranlib" \
    STRIP=${target_host}-strip \
    PKG_CONFIG_PATH="/usr/${target_host}/lib/pkgconfig" \
    PKG_CONFIG_SYSROOT_DIR="/usr/${target_host}" \
    LIB_SYS_TRY_SHARED=1
mkdir -p .cargo && \
                echo "[target.${target_host_rust}]" > .cargo/config.toml && \
                echo "linker = \"${target_host}-gcc\"" >> .cargo/config.toml && \
                echo "ar = \"${target_host}-ar\"" >> .cargo/config.toml && \
                echo "" >> .cargo/config.toml && \
                echo "[patch.crates-io]" >> .cargo/config.toml && \
                echo "libz-sys = { path = '/tmp/libz-sys' }" >> .cargo/config.toml && \
                cargo build --target="${target_host_rust}" --release

with zlib manually installed, just in case - and this in a Debian image:

__ZLIB_VERSION='1.2.13' && \
      ( cd /tmp && \
          wget -O /tmp/zlib-${__ZLIB_VERSION}.tar.gz https://zlib.net/fossils/zlib-${__ZLIB_VERSION}.tar.gz && \
          tar xf zlib-${__ZLIB_VERSION}.tar.gz && \
          rm zlib-${__ZLIB_VERSION}.tar.gz )

and I patch libz-sys by:

      ( cd /tmp && git clone https://github.com/rust-lang/libz-sys ) && \
      ( cd /tmp/libz-sys && wget -qO- 'https://github.com/rust-lang/libz-sys/pull/206.patch' | patch -p1 )

but it seems to not understand, and the aarch64 compiler is called with invalid flags

The following warnings were emitted during compilation:                                                                                                                                                                                                                                                                                                                                                 
warning: [email protected]: aarch64-linux-gnu-gcc: error: unrecognized command-line option ‘-m64’
warning: [email protected]: ToolExecError: Command "aarch64-linux-gnu-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "src/zlib" "-fvisibility=hidden" "-DSTDC" "-D_LARGEFILE64
_SOURCE" "-D_POSIX_SOURCE" "-o" "/boltz-backend/boltzr/target/release/build/libz-sys-2e1975913bc64b22/out/lib/0dc752f03a07a721-adler32.o" "-c" "src/zlib/adler32.c" with args aarch64-linux-gnu-gcc did not execute successfully (status code exit status: 1).cargo:warning=aarch64-linux-gnu-gcc: error: unrecognized command-line option ‘-m64’

The project I'm trying to cross-compile (from amd64) is this one: https://github.com/BoltzExchange/boltz-backend.git

@Byron
Copy link
Member

Byron commented Jan 18, 2025

Maybe a different issue? The local override can also point to a local checkout of the crate, which might help to make adjustments to build.rs for a fix. Maybe something that can eventually be fully understood and contributed.

@tsjk
Copy link

tsjk commented Jan 18, 2025

Yes, this was related to something else. I managed to cross-compile by overriding the dependency. Thanks. :)

@polarathene
Copy link
Contributor

polarathene commented Jul 2, 2025

UPDATE: PR: #244

TL;DR: let cross_compiling = target != host appears to be a redundant legacy and opinionated condition.

The correct solution would be to remove it entirely, since contributions after Feb 2016 have made that much more viable:

  • Given skipping the forced build will attempt to search for zlib and if not found, fallback to building from source anyway.
  • Anyone that wants to force static linking can do so via the static feature or LIB_ZSYS_STATIC=1 ENV. However, the build environment should still support opt-out if linking is preferable despite a transitive dependency choosing to opt-in to static.

NOTE: Remainder of this response is for reference/context to justify the change, with an abundance of links for traceability.


The forced build for cross-compilation was added in Feb 2016 with this Sep 2018 PR adding the following comment:

cross compiling almost never has a prebuilt version.

Which is presumably more about building for a different OS, rather than architecture given the target conditions? (and technically musl/glibc targets on linux). Having attempted cross-compiling for a foreign arch in the past I do recall that having more friction, but I'd chalk that up partially to inexperience and tooling options available at the time.

In Mar 2016 want_static / LIBZ_SYS_STATIC=1 ENV opt-in to force static builds for other scenarios (in Oct 2018 the associated static feature was introduced). Since then there's the current targets conditions (and prior targets previously handled like musl, android, and another windows target), while since the Sep 2018 PR we now just force dynamic linking for certain targets like android:

libz-sys/build.rs

Lines 25 to 31 in 9c453d3

// All android compilers should come with libz by default, so let's just use
// the one already there. Likewise, Haiku and OpenHarmony always ship with libz,
// so we can link to it even when cross-compiling.
if target.contains("android") || target.contains("haiku") || target.ends_with("-ohos") {
println!("cargo:rustc-link-lib=z");
return;
}

After potentially building zlib, there's even a function to test searching for zlib library to link (implemented in Sep 2018 too from this earlier discussion + later related fix for debug builds (Feb 2024) bundled with a cross-compile fix of a fix for the apple target exception):

libz-sys/build.rs

Lines 93 to 102 in 9c453d3

// If we've gotten this far we're probably a pretty standard platform.
// Almost all platforms here ship libz by default, but some don't have
// pkg-config files that we would find above.
//
// In any case test if zlib is actually installed and if so we link to it,
// otherwise continue below to build things.
if zlib_installed(&mut cfg) {
println!("cargo:rustc-link-lib=z");
return;
}

That'd be valid in a cross-compilation scenario and verify the library is available 🤔 The cross_compiling constraint seems redundant/legacy?

I guess it's a bit late to address this concern without the risk of some breakage? But since not all -sys crates are going to have such behaviour with cross-compilation detection, this forced build behaviour doesn't really address the concern it's trying to assist with? (Plus as shown in my follow-up comment it can be perfectly valid to build for a foreign arch where you have a library available to link, so it should be supported)

For consistency regarding cross-compiling, when would it make sense for someone to build for their native arch and foreign arch, but have the linking differ?

  • This constraint was added before the LIBZ_SYS_STATIC=1 support was added.
  • Likewise there is a fallback (if the cross-compile detection was removed to avoid that early return), that would test if zlib could be found for linking, otherwise build it anyway.

If I am building and want dynamic linking or static linking, I'd want consistency (except when it's not viable), not troubleshooting why builds cross-compiled aren't linking as expected, but are when I run them through QEMU / native host for that arch.


Your build environment should not need to be a native arch for linking to be supported, remove the constraint entirely? Those that want static builds can do so via the ENV that's been available since 2016.

Features can be useful for crates as was discussed earlier in this PR feedback by @jongiddy , but when transitive deps do this it can be problematic especially as an end user building a project/crate and you want to have control over your build. ENV is much more useful there so I agree that LIBZ_SYS_STATIC=0 should cancel out a crate that has enabled the static feature (just like many other crates have been mentioned to already support).

The proposal to leverage a feature for opt-in that conflicts with static is fine, you'll see cudarc has similar use of features to choose dynamic vs static linking, but that can be a tad awkward to enforce a decision via feature flag when it shouldn't need to.

  • You still run into the situation of ugly workarounds when the feature decision was via a transitive dependency out of your control.
  • Using LIBZ_SYS_STATIC=0 would work better and for top-level projects where using an ENV is actually relevant (and has other advantages), this can be committed into .cargo/config.toml which addresses the earlier caveat raised about hidden dependency as it would be standard Cargo config location, not unlike Cargo.toml where the workaround via feature is more of a code smell.

@polarathene
Copy link
Contributor

polarathene commented Jul 2, 2025

@tsjk you can more easily cross-compile via Zig (with cargo-zigbuild for Rust support).

Here's a full example within a Docker container (Fedora 42) as an example, but you should be able to adapt to your equivalent environment rather easily:

# Fedora container for a reproducible build environment, running on an amd64 / x86_64 host:
$ docker run --rm -it --workdir /example fedora:42

# Install base deps:
# Boltz specific deps needed to compile from host arch: npm + protobuf-compiler
dnf install -yq gcc git-core npm protobuf-compiler rustup zig

# Setup Rust with Zig:
rustup-init -y --profile minimal --default-toolchain stable
export PATH="/root/.cargo/bin:${PATH}"
cargo install cargo-zigbuild

# Add cross-compilation support (libs needed for linking + arm64 target):
# NOTE: `openssl-sys` additionally needs to parse headers and since the host vs target arch differs,
# you will need to use `OPENSSL_DIR` or `PKG_CONFIG_SYSROOT_DIR` ENV.
export SYSROOT_ARM64=/opt/sysroot/aarch64-gnu
dnf --installroot ${SYSROOT_ARM64} --use-host-config --setopt=install_weak_deps=0 --forcearch=aarch64 \
  install -yq libpq-devel openssl-devel
rustup target add aarch64-unknown-linux-gnu

# Grab the project source:
git clone --recurse-submodules https://github.com/BoltzExchange/boltz-backend .

# Rust build relies upon files generated from `node_modules`:
# NOTE: Using lock file fails due to outdated `[email protected]` requirement, where Python 3.13 no longer providing a distutils module
npm install --no-package-lock

# Now you can build the rust project for the foreign arch:
# - `PKG_CONFIG_SYSROOT_DIR` is only needed for cross-compilation for `openssl`,
#   alternatively skip the need for `pkg-config` via `OPENSSL_DIR="${CROSS_ROOT_ARM64}/usr"`
# - `RUSTFLAGS` must be set to the aarch64 sysroot for correctly linking to the arm64 `.so` files
cd boltzr
PKG_CONFIG_SYSROOT_DIR="${SYSROOT_ARM64}" RUSTFLAGS="-L ${SYSROOT_ARM64}/usr/lib64" cargo zigbuild --release --target aarch64-unknown-linux-gnu

I think that is simpler than using Cargo to cross-compile without Zig 😅

The PR was not used in my example for a successful build via Zig, so it would have built from source successfully? (EDIT: Seems that it was a build-dep, whereas your linked libz for OpenSSL is runtime, thus no conflict)


Verification:

libz is linked:

# Required as not running via `chroot` or via an aarch64 container/host:
# - `ld-linux-aarch64.so.1 --list` is the equivalent to `ldd` (which is often a shell script wrapper).
# - `LD_LIBRARY_PATH` to add libs from the aarch64 sysroot to the search path for resolving libs.
$ LD_LIBRARY_PATH=/opt/sysroot/aarch64-gnu/usr/lib64 /opt/sysroot/aarch64-gnu/lib/ld-linux-aarch64.so.1 --list target/aarch64*/release/boltzr

        linux-vdso.so.1 (0x0000400000810000)
        libpq.so.5 => /opt/sysroot/aarch64-gnu/usr/lib64/libpq.so.5 (0x0000400003d30000)
        libssl.so.3 => /opt/sysroot/aarch64-gnu/usr/lib64/libssl.so.3 (0x0000400003da0000)
        libcrypto.so.3 => /opt/sysroot/aarch64-gnu/usr/lib64/libcrypto.so.3 (0x0000400003e90000)
        libm.so.6 => /opt/sysroot/aarch64-gnu/usr/lib64/libm.so.6 (0x0000400004290000)
        libpthread.so.0 => /opt/sysroot/aarch64-gnu/usr/lib64/libpthread.so.0 (0x0000400004360000)
        libc.so.6 => /opt/sysroot/aarch64-gnu/usr/lib64/libc.so.6 (0x0000400004390000)
        libdl.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libdl.so.2 (0x0000400004560000)
        /lib/ld-linux-aarch64.so.1 => /opt/sysroot/aarch64-gnu/lib/ld-linux-aarch64.so.1 (0x00007cf124eb0000)
        libgssapi_krb5.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libgssapi_krb5.so.2 (0x0000400004590000)
        libldap.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libldap.so.2 (0x0000400004600000)
        libz.so.1 => /opt/sysroot/aarch64-gnu/usr/lib64/libz.so.1 (0x0000400004690000)
        libkrb5.so.3 => /opt/sysroot/aarch64-gnu/usr/lib64/libkrb5.so.3 (0x00004000046d0000)
        libk5crypto.so.3 => /opt/sysroot/aarch64-gnu/usr/lib64/libk5crypto.so.3 (0x00004000047b0000)
        libcom_err.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libcom_err.so.2 (0x00004000047f0000)
        libkrb5support.so.0 => /opt/sysroot/aarch64-gnu/usr/lib64/libkrb5support.so.0 (0x0000400004820000)
        libkeyutils.so.1 => /opt/sysroot/aarch64-gnu/usr/lib64/libkeyutils.so.1 (0x0000400004850000)
        libresolv.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libresolv.so.2 (0x0000400004880000)
        liblber.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/liblber.so.2 (0x00004000048b0000)
        libevent-2.1.so.7 => /opt/sysroot/aarch64-gnu/usr/lib64/libevent-2.1.so.7 (0x00004000048f0000)
        libsasl2.so.3 => /opt/sysroot/aarch64-gnu/usr/lib64/libsasl2.so.3 (0x0000400004970000)
        libselinux.so.1 => /opt/sysroot/aarch64-gnu/usr/lib64/libselinux.so.1 (0x00004000049b0000)
        libcrypt.so.2 => /opt/sysroot/aarch64-gnu/usr/lib64/libcrypt.so.2 (0x0000400004a00000)
        libpcre2-8.so.0 => /opt/sysroot/aarch64-gnu/usr/lib64/libpcre2-8.so.0 (0x0000400004a50000)

But as patchelf shows, not directly, it is via the linked OpenSSL dep:

# Alternatively... chroot in:
$ cp target/aarch64*/release/boltzr /opt/sysroot/aarch64-gnu/tmp/boltzr
$ dnf --installroot /opt/sysroot/aarch64-gnu --use-host-config --forcearch=aarch64 install patchelf
$ chroot /opt/sysroot/aarch64-gnu

# Direct deps linked (zlib from ldd was implicit from openssl link):
$ patchelf --print-needed /tmp/boltzr
libpq.so.5
libssl.so.3
libcrypto.so.3
libm.so.6
libpthread.so.0
libc.so.6
libdl.so.2
ld-linux-aarch64.so.1

$ patchelf --print-needed /usr/lib64/libcrypto.so.3
libz.so.1
libc.so.6
ld-linux-aarch64.so.1

# `libcrypto` is the only dep requiring `libz`:
$ exit
$ cargo install lddtree
$ lddtree target/aarch64*/release/boltzr /opt/sysroot/aarch64-gnu | grep -B1 libz
# Output omitted

# Here we can see that boltzr project uses `libz-sys` crate as a build-dep:
$ cargo tree -i libz-sys
libz-sys v1.1.20
└── libgit2-sys v0.18.0+1.9.0
    └── git2 v0.20.0
        └── built v0.8.0
            [build-dependencies]
            └── boltzr v3.11.0 (/example/boltzr)

So in your case libz-sys is specifically built as a build-dep.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Link against shared zlib library when cross-compiling
7 participants