Building Rust from source with `x.py dist` uses binary components

[paulmenzel]

In issue #86436, @Mark-Simulacrum commented:

FWIW we don't generally expect people to use x.py build in dist tarballs -- x.py dist/install are more likely to work.

I tested ./x.py dist with Rust 1.56.1, but unfortunately, it does not build every component from source, but downloads rust-std, rustc and cargo 1.55.0, containing the binaries.

[…]
+ python3 ./x.py dist
info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default.
downloading https://static.rust-lang.org/dist/2021-09-09/rust-std-1.55.0-x86_64-unknown-linux-gnu.tar.xz
######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/rust-std-1.55.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-09/rustc-1.55.0-x86_64-unknown-linux-gnu.tar.xz
######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/rustc-1.55.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-09/cargo-1.55.0-x86_64-unknown-linux-gnu.tar.xz
######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/cargo-1.55.0-x86_64-unknown-linux-gnu.tar.xz
   Compiling proc-macro2 v1.0.24
[…]

Can the already installed Rust in the system be used instead?

Otherwise, x.py build should be the supported way for distributions to build Rust.

[paulmenzel]

The Cargo archive contains cargo-1.55.0-x86_64-unknown-linux-gnu/cargo/bin/cargo.

[the8472]

Rustc is self-hosted, which means it needs a rust compiler to build itself so by default it downloads the previous version.
If your distro already has a one-older rust compiler you can use that instead by adjusting your config.toml

See the rustc-dev guide for an explanation of the build process.

If you want to bootstrap from a C compiler you can start with mrustc, e.g. that's what guix does (they have updated their workflow since that blogpost).

[paulmenzel]

Thank you. We only have Rust 1.52.1 in our distribution (MarIuX), so we stay with ./x.py build for now.

[paulmenzel]

Hmm, ./x.py build also downloaded the Rust 1.55 binaries. :(

+ python3 ./x.py build --exclude src/tools/miri
info: looks like you are running this command under `sudo`
      and so in order to preserve your $HOME this will now
      use vendored sources by default.
downloading https://static.rust-lang.org/dist/2021-09-09/rust-std-1.55.0-x86_64-unknown-linux-gnu.tar.xz
^M######                                                                     9.5%^M##################################################                        70.0%^M######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/rust-std-1.55.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-09/rustc-1.55.0-x86_64-unknown-linux-gnu.tar.xz
^M                                                                           0.4%^M########                                                                  12.2%^M######################                                                    31.1%^M############################################                              61.2%^M######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/rustc-1.55.0-x86_64-unknown-linux-gnu.tar.xz
downloading https://static.rust-lang.org/dist/2021-09-09/cargo-1.55.0-x86_64-unknown-linux-gnu.tar.xz
^M##########################                                                36.8%^M######################################################################## 100.0%
extracting /pkg/rustc-1.56.1-0/build/rustc-1.56.1-src/build/cache/2021-09-09/cargo-1.55.0-x86_64-unknown-linux-gnu.tar.xz
Compiling proc-macro2 v1.0.24
[…]

(From a cursory glance, I do not see a difference to ./x.py dist.)

[the8472]

build builds individual executables and libraries into the ./build output directory. dist then produces tarballs. And yes it is the build step that requires the previous rustc version.

[paulmenzel]

After working around the problems described in issue #90562, the build fails when using Rust 1.52.1.

[build]
cargo = "/pkg/rustc-1.52.1-0/bin/cargo"
rustc = "/pkg/rustc-1.52.1-0/bin/rustc"
rustfmt = "/pkg/rustc-1.52.1-0/bin/rustfmt"

Now building:

export PATH=/pkg/rustc-1.52.1-0/bin:$PATH LD_LIBRARY_PATH=/pkg/rustc-1.52.1-0/lib:$LD_LIBRARY_PATH RUSTFLAGS="$RUSTFLAGS -C link-args=-lffi"  &&
python3 ./x.py build --exclude src/tools/miri
[…]
   Compiling toml v0.5.7
error[E0277]: `[&str; 2]` is not an iterator
    --> src/bootstrap/compile.rs:1140:27
     |
1140 |             for flavor in ["ld", "ld64"] {
     |                           ^^^^^^^^^^^^^^ borrow the array with `&` or call `.iter()` on it to iterate over it
     |
     = help: the trait `Iterator` is not implemented for `[&str; 2]`
     = note: arrays are not iterators, but slices like the following are: `&[1, 2, 3]`
     = note: required because of the requirements on the impl of `IntoIterator` for `[&str; 2]`
     = note: required by `into_iter`

error[E0277]: `[&str; 2]` is not an iterator
   --> src/bootstrap/dist.rs:415:31
    |
415 |                 for flavor in ["ld", "ld64"] {
    |                               ^^^^^^^^^^^^^^ borrow the array with `&` or call `.iter()` on it to iterate over it
    |
    = help: the trait `Iterator` is not implemented for `[&str; 2]`
    = note: arrays are not iterators, but slices like the following are: `&[1, 2, 3]`
    = note: required because of the requirements on the impl of `IntoIterator` for `[&str; 2]`
    = note: required by `into_iter`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0277`.
error: could not compile `bootstrap`

No idea, why toml is required to build a rustc 1.56.1.

[the8472]

Like I said, it requires a one-older rust, the previous version. I.e. Building 1.56 requires 1.55
If you're starting from 1.52 you'll have to chain to 1.53, 1.54, 1.55 and then you can build 1.56.