Installing the ocamllsp dev tool rebuilds the compiler unnecessarily

[gridbugs]

If the compiler is already installed as a dependency of the current project, installing ocamllsp should not require building the compiler a second time. Instead it should re-use the compiler that was installed as a dependency of the current project.

However currently installing ocamllsp as a dev tool causes the compiler to be rebuilt:

$ dune init proj foo . 
Success: initialized project component named foo
$ ls
_build          bin             dune-project    foo.opam        lib             test
$ dune pkg lock 
Solution for dune.lock:                                              
- arch-x86_64.1
- conf-mingw-w64-gcc-i686.1
...
- ocaml-env-mingw64.1
- system-mingw.1
$ dune build 
 Downloading ocaml-compiler.5.3.0
    Building ocaml-compiler.5.3.0
    Building ocaml-base-compiler.5.3.0
$ dune build

# We have to rebuild the whole compiler from scratch, even tho we are using the same compiler
$ dune tools install ocamllsp
Solution for dev-tools.locks/ocaml-lsp-server:                                              
- astring.0.8.5
- base.v0.17.3
...
- uutf.1.0.4
- xdg.3.20.2
- yojson.2.2.2
    Building base-unix.base                         
    Building ocaml-compiler.5.3.0 
    Building ocaml-base-compiler.5.3.0
    Building ocaml-config.3       
    Building ocaml.5.3.0          
    Building seq.base              
...           
    Building dune-rpc.3.20.2       
    Building ocaml-lsp-server.1.23.0

(copied from #12244 (comment))

This happens when using portable lockdirs and non-portable lockdirs. I've reproduced this on 3.20.2 and also the latest nightly build at the time of writing (d9fc1d3).

[gridbugs]

Dune is creating these two directories in the toolchains directory:

$ ls -1 ~/.cache/dune/toolchains/
ocaml-compiler.5.3.0-9455b2385cca98d59134c6a422e63f22
ocaml-compiler.5.3.0-a3dd2a36d935a7ada272441c8543eba1

Dune appends hashes of some package metadata to the end of the directory name since sometimes it is necessary to build the same version of the compiler multiple times (e.g. due to optional dependencies being available). However that shouldn't be happening here. Either some metadata is being included in the hash that shouldn't be, or something is meaningfully different about the environment where ocamllsp is being built from the project itself (in which case we should remove that difference).

[gridbugs]

I noticed the hashes in toolchain dir names were computed with Digest.generic which I think is not safe to use on sets or maps since two values which are considered equal may have different in-memory representations. I changed it to hash with Lock_dir.Pkg.hash which avoids this problem and now only a single toolchains directory is created.

However ocaml-compiler is still rebuilt when installing ocamllsp, so this alone did not solve the problem. I'll keep investigating.

[gridbugs]

I looked further into this today. I had a hunch it was related to the extra-sources field adding additional files to the target directory and causing the cookie for the compiler in the project's deps to differ from the compiler in ocamllsp's deps, but removing extra-sources such that the cookies match doesn't solve the problem. I don't totally understand how dune decides whether to used the cached copy of an artefact rather than rebuilding it, but learning more about how that works is my next step here. (Although the compiler itself won't be in dune's cache, dune still must be caching something on behalf of the compiler since dune build; dune clean; dune build does not rebuild the compiler on the second dune build.)

Possibly related to #12390

[gridbugs]

I think this will be solved by #12179.