Pull out the HTTP mock server as `cargo-dev-registry`

[0xPoe]

Problem

In Cargo's unit tests, we used a mock HTTP server as our test infrastructure. But it is not a fully implemented registry. Some unit tests are just using custom handler to mock some registry API.

So as I discussed with @weihanglo and @epage, we both agree that having full featured dev registry in cargo would help the testing a lot. And maybe help us to catch some bugs that can only be caught in a real cargo registry.

At the same time, if we can have a standalone dev registry, it would help us a lot when testing or debugging the cargo itself. At the moment, it is difficult to test the real publish function, because setting up a real registry from scratch is difficult.

Furthermore, if we try to implement this dev registry, we can give other people a reference about how to implement the basic API for a cargo registry. We can also extend the common library for building a cargo registry. We believe this would not only help the cargo itself, but also help the community to implement a registry more easily.

Proposed Solution

As discussed with @epage, he suggested that we can do the following things to pull out the mock HTTP server as our cargo-dev-registry.

• switch cargo's summary generation to use a normalized manifest, rather than a Package
• pull registry types like IndexPackage out into cargo-util-schemas
• pull out index/manifest/publish conversions into a cargo-util-registry
• switch our dev registry to using the above rather than reimplement
• pull our dev registry out into cargo-dev-registry with an optional cli for spinning it up

Notes

No response

[0xPoe]

@arlosi If I remember correctly, I think we mentioned in one of our weekly meetings that we wanted a full-featured registry to test the Cargo, but I don't remember the context. Maybe it was related to authentication. Do you remember that requirement or use case?

[epage]

For past references, see

• https://rust-lang.zulipchat.com/#narrow/channel/246057-t-cargo/topic/Downloading.20a.20crate.20and.20all.20of.20its.20dependencies/near/466097168
• https://hachyderm.io/@epage/112893814784038149

CC @shepmaster, @Turbo87 as this involves trying to share more code between Cargo and registries

As discussed with @epage, he suggested that we can do the following things to pull out the mock HTTP server as our cargo-dev-regsitry.

To clarify, I expect cargo-dev-regsitry to be a first-party crate, not a built-in command.

pull out index/manifest/publish conversions into a cargo-util-registry

Do we want to start with cargo-util-registry and cargo-dev-registry being experimental, internal, or intentional artifacts? See the policy and our charter which makes it only an FCP and not an RFC.

[epage]

In addition to accepting this, some thoughts when we discussed this as a team

• Having cargo-dev-registry serve both to show how a registry works and as a dev-registry might come into conflict with each other because we might want test-specific features, like failure injections. We'll initially prioritize it as a dev-registry and then see if it can also serve as an example
• As such, we might mark cargo-dev-registry as experimental. However, we didn't see a reason to do the same for cargo-util-registry
• We considered having crates-io own the cargo-util-registry logic but crates-io is mostly client focused and putting server oriented logic in it might make it not serve its purpose as well for both, particularly in the server getting large client dependencies (and the name is specific to one registry)
• While cargo-util-registry may have logic to help with registries, not all may find it useful
  • e.g. converting from a Manifest to an Index won't help crates.io because they have an intermediate step of putting things in a database iiuc
  • e.g. https://docs.rs/reg-index/latest/reg_index/
• If we're able to create conformance tests or have this operate as a test double, that could also be usefl

[0xPoe]

@rustbot claim

[0xPoe]

• switch cargo's summary generation to use a normalized manifest, rather than a Package

@epage Do you mean to change the Summary::new to take a normalized manifest as the the only parameter?

[epage]

Summary is a higher-level type.

I'm referring to IndexPackage being created from a normalized TomlManifest

cargo/src/cargo/sources/registry/index/mod.rs

Lines 197 to 199 in ef5edff

	/// A single line in the index representing a single version of a package.
	#[derive(Deserialize, Serialize)]
	pub struct IndexPackage<'a> {

We then convert these into a Summary

cargo/src/cargo/sources/registry/index/mod.rs

Lines 723 to 747 in ef5edff

	let IndexPackage {
	name,
	vers,
	cksum,
	deps,
	mut features,
	features2,
	yanked,
	links,
	rust_version,
	v,
	} = serde_json::from_slice(line)?;
	let v = v.unwrap_or(1);
	tracing::trace!("json parsed registry {}/{}", name, vers);
	let pkgid = PackageId::new(name.into(), vers.clone(), source_id);
	let deps = deps
	.into_iter()
	.map(|dep| dep.into_dep(source_id))
	.collect::<CargoResult<Vec<_>>>()?;
	if let Some(features2) = features2 {
	for (name, values) in features2 {
	features.entry(name).or_default().extend(values);
	}
	}
	let mut summary = Summary::new(pkgid, deps, &features, links, rust_version)?;

Now that I think of it, I'm unsure if we ever create an IndexPackage from a manifest, except in cargo-test-support and doubt its worth going from TomlManifest -> IndexPackage -> Summary unless we are doing it to cache non-local transitive dependencies for faster resolving but then we'll likely need to parse the full manifest anyways.