Include all shipped files in the manifest

[pietroalbini]

Currently, the manifests don't include all the files available in the release, meaning that promote-release has to whitelist extra paths to preserve in its pruning code (introduced by rust-lang/rust#78196 and #21).

Namely, they don't include the rustc-{channel}-src.{ext} tarballs. Those tarballs are problematic since they are not installable through rustup, as they don't follow the standard format (for example here is the rust-src component.

rust-src-1.47.0.tar.xz
├── components
├── install.sh
├── rust-installer-version
└── rust-src
    ├── lib
    │   └── rustlib
    │       └── src
    │           └── rust
    │               ├── Cargo.lock
    │               └── library/
    └── manifest.in

rustc-1.47.0-src.tar.xz
├── Cargo.lock
├── Cargo.toml
├── config.toml.example
├── configure
├── CONTRIBUTING.md
├── COPYRIGHT
├── git-commit-hash
├── library/
├── LICENSE-APACHE
├── LICENSE-MIT
├── README.md
├── RELEASES.md
├── src/
├── vendor/
├── version
└── x.py

A way to solve this issue cleanly would be to figure out a way to include the rustc-{channel}-src tarballs in the manifest, thus making sure that all files we ship are referenced in a single manifest file. The best way to include those files is still an open question.

Once this is fixed this snippet can be removed.

[pietroalbini]

@Mark-Simulacrum proposed to treat rustc-{channel}-src as a component with a new installable = false key, but that might mess up old rustup versions (cc @kinnison). Another idea I had was to create an artifacts section of the manifest, which would be currently ignored by rustup (logic for it could be added later if people need it).

[Mark-Simulacrum]

Hm, so the artifacts idea seems even better, but I do worry a bit that it loses the nice-ish ability to provide signatures and various compression formats for a single artifact without listing it as distinct artifacts. Maybe that's not too important though, and in practice given we just have the src tarballs today... seems fine to just have a table for now.

[pietroalbini]

@Mark-Simulacrum oh, I was thinking of using the same schema we have right now, but in a different top-level map:

[artifacts.rustc-src.target."*"]
available = true
url = "https://static.rust-lang.org/dist/2020-10-27/rustc-dev-nightly-x86_64-pc-windows-msvc.tar.gz"
hash = "4b2ed13b88ba98b853c7f46034f3c7424e05caf7fe0c9b52ca93d90894d39285"
xz_url = "https://static.rust-lang.org/dist/2020-10-27/rustc-dev-nightly-x86_64-pc-windows-msvc.tar.xz"
xz_hash = "731b8c0d37dfadad3ac7187e3832ee51c25a0cd40caa1faf45297b3960abbfe0"

...but that is less flexible, and doesn't support uncompressed stuff.

[kinnison]

Rustup determines if a component is available based on whether it has the url/hash pairs and the available key. I'm not sure what it'd do with a component which has urls but had available = false. The toml is parsed here: https://github.com/rust-lang/rustup/blob/master/src/dist/manifest.rs#L419 which suggests it'd just ignore the url etc. if the available key is set to false, but I'd not stake a bet on it without some proper testing which I'm not in a position to do right now.

If, on the other hand, you add a new table then that'll simply be ignored by rustup until such time as we decide if/how we want to handle them.

If you do, then I strongly suggest you use a better scheme than the above though, something like:

[[artifacts.rustc-src.target."*"]]
url = "https://...."
hash = "439438209489032..."

Might be better, since compression type can be detected at runtime easily enough, and that way we don't need to change the channel format for different kinds of artifact.

[pietroalbini]

@kinnison that list approach is really interesting, especially for files that are not tarballs: in #23 I discovered that we also need to whitelist .msi and .pkg files for the Windows and macOS installers.

[pietroalbini]

Fixed by rust-lang/rust#78486.