buildRustPackage is not very nix-ish

[FlorentBecker]

Hi, I started developping some rust code on nixos, and I find that while buildRustPackage is great for having all of crates.io available, the experience is not quite on par with, say, that of developping in haskell or ocaml on nix.

Specifically:

• Documentation is a bit terse (I only found this wiki page: https://nixos.org/wiki/Rust), and it is not clear if it is up to date.
• In Rust projects with both Rust and C dependencies, having the C dependencies known of nix, and the Rust dependencies known to Cargo is not convenient: Cargo.lock cannot be generated from within a nix-shell because of depsSha256, so one needs the awkward nix-shell -p cargo -p rustc -p
• C dependencies of Rust dependencies of a project need to be discovered and added manually to default.nix (dependencies are forgotten when going through Cargo-land)
• When using nix-shell, builds of rust dependencies are not shared between rust projects.

I'm grouping these into one issue because I'm not sure which of these sub-issues can be fixed in buildRustPackage, and what would need something more akin to cabal2nix where rust dependencies of rust packages are made manifest to nix/within nixpkgs.

[wizeman]

I agree with your points.

Regarding your 3rd point, that may not be very easy to accomplish. I was thinking we'd need something that parses Cargo.toml, but even Cargo.toml doesn't seem to tell you what C dependencies you need, see for example: https://github.com/brson/rust-sdl/blob/master/Cargo.toml
This may be even harder to figure out when libraries are dynamically loaded via dlopen. Case in point, the x11-dl library, which loads its dependencies dynamically: https://github.com/Daggerbot/x11-rs/blob/master/x11-dl/Cargo.toml

If I understand correctly your 4th point, I think it may also not be easy to accomplish at this point in time (but I hope I'm wrong). As far as I know, there is no way to tell cargo how to use an already-built library. I think I asked the Cargo developers how to do this, and IIRC they told me this is not supported because Rust doesn't have a stable ABI yet (which is probably a poor excuse in Nix's case).

[cstrahan]

Regarding the 3rd point, perhaps we could have something that crawls a source tree looking for stuff like #[link(name="SDL")], and then we just need a map from the SO name to the Nixpkgs package name. Perhaps a similar thing could be done for dlopen (at least where dlopen is used with a string litteral).

That might pull in some false positives, but then it's just a matter of removing the unnecessary stuff, which is much easier IMO than trying to hunt down the dependencies in the first place.

[cstrahan]

And regarding the 4th point, as @wizeman notes:

If I understand correctly your 4th point, I think it may also not be easy to accomplish at this point in time (but I hope I'm wrong). As far as I know, there is no way to tell cargo how to use an already-built library. I think I asked the Cargo developers how to do this, and IIRC they told me this is not supported because Rust doesn't have a stable ABI yet (which is probably a poor excuse in Nix's case).

That's precisely what I recall from the last time I looked into Rust on Nix.

[Ericson2314]

As far as I know, there is no way to tell cargo how to use an already-built library.

I'd make these two simplish requests of them:

• Allow configuring/building dependencies with a downstream lockfile
• Allow an unstable way to pass arbitrary flags to rustc without cargo complaining.

Then we should be to do the rest ourselves.

[FlorentBecker]

Allow configuring/building dependencies with a downstream lockfile

Don't overrides allow to do that?

[retrry]

I think the whole rust situation is bound to become worse for some time. I was trying to update cargo to 0.10.0 version, but it breaks buildRustPackage function.
-- Cargo needs write access to registry as they implemented file locking (rust-lang/cargo#2486). And it seems that they now try to write .cargo-lock-index file to registry directory. This can be mitigated by copying whole registry to package build directory with write permissions.
-- Cargo fetch now barfs at NixOS hack with file:///dev/null as it seems that they always try to update registry.

    Updating registry `file:///dev/null`
error: failed to fetch `file:///dev/null`

Caused by:
  [6/-1] Could not find repository from '/dev/null'

If somebody could look into this, as I don't have enough knowledge and experience with this kind of stuff :) Or maybe we should wait for: rust-lang/cargo#2361

[Mic92]

Could cargo-local-registry be used here to improve the situation?

[Ericson2314]

Not very much :(. The problem is caching rlibs, not source.

[FlorentBecker]

Could we use multiple output and have each rust package output both its binaries / rlibs and its source. Then each rust package in nix could depend on the "source" output of its cargo dependencies. I think this would solve issues 2 and 3 of my original report, since rust-to-rust dependencies would be tracked by nix.

[ckauhaus]

See this Reddit discussion + blog post for an alternative approach. Looks interesting.

[8573]

And the relevant nixpkgs issue ticket: #24991.