Problems with the planned new cross-compilation model

[finagolfin]

Description

@compnerd put together a document on a planned new cross-compilation model last summer, which replaces the flags -sdk and -resource-dir with the flags -sdk and -sysroot instead. I've since seen some issues with this new model, in order of importance:

1. Explicitly specifying an -sdk has a bunch of bugs and doesn't work very well. The most common way to ship Swift core modules and runtime libraries is next to the compiler instead and explicitly specifying an external -resource-dir works reasonably, though bugs have crept in there too. However, explicitly specifying an external -sdk, particularly a full SDK that contains both a C/C++ sysroot and a Swift resource directory, has historically been the least used configuration and comes with a lot of bugs.
   For example, I just saw that the Windows CI tries to build the Foundation macros for the Windows host by using a 6.0.3 Windows SDK with the trunk 6.2 Swift compiler. As explained there, that works fine now because the Swift 6.2 compiler likely quietly ignores the 6.0.3 resource directory in that 6.0.3 SDK and uses the 6.2 stdlib modules next to the compiler instead, but when I enforced that it must use the 6.0.3 SDK alone in that pull, the compile failed as expected. We will need to shake out all such -sdk leaks before we can rely on this flag for cross-compilation.
2. The -sdk/-resource-dir flags look directly in those paths for the C sysroot and Swift runtime resources, but when using -sdk/-sysroot instead, -sdk looks in <sdkPath>/usr/lib/swift instead for the Swift-specific files. I understand why this was done, for backward compatibility of the -sdk flag, but this means you cannot specify an arbitrary Swift resource path anymore.
   For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.
3. The -sysroot flag is brand new and barely used, I see a couple bugs in its implementation already. This is easiest to remedy since it's new.

Regarding 1., since all current approaches use -sdk, we have no choice but to make it better. My in-progress work on making it work properly in #79621 is shaking out more bugs, which I will file and fix as I'm able. At the very least, this suggests we might have to go slower with cross-compilation SDKs while these problems are shaken out.

@compnerd, let me know your thoughts.

@al45tair, as PSG chair, maybe you have some input.

@etcwilde, you deal with these cross-compilation issues more than most, thoughts welcome.

@MaxDesiatov, the SDK bundles feature you helped create uses the current -sdk/-resource-dir flags a lot, perhaps you have some input.

[etcwilde]

So I don't think getting rid of the -resource-dir is the goal of that proposal, though certainly lessening the reliance on setting it. The compiler resources are meant to be things that are tied to a given version of the compiler. Swift hasn't been good about keeping them separate, so I'll use clang as an example, but the sanitizer runtimes and the clang builtin headers are specific to that exact clang and thus must ship as part of the toolchain. That's what the -resource-dir should point at. For Swift, we definitely want the clang-importer headers in a subdirectory of the resource directory. Some of the shims also probably belong in that directory too like HeapObject.h and Task.h, but not all of them. The C++ bridging code isn't stable and is also tied to a specific build of the compiler, so those should also be shipped in there too. Under normal circumstances, people shouldn't really need to touch this, but stuff happens.

Under the new definition, the -sdk flag should point to the Swift runtimes for the platform you're targeting, and then the -sysroot should point at the runtimes and libraries provided by the OS. This is mostly because most OS's don't come with a Swift stdlib and don't want to install the Swift runtimes into your sysroot, or can't because you're trying to build something for your local system and don't have privileges to do so.

I haven't personally lived on the flags yet since we haven't split stuff up and things aren't done cooking in this regard yet, but this will become more important as we split the SDK distribution from the per-platform toolchain monolith that we have today. My eventual goal is to be able to download an Ubuntu sysroot, an Ubuntu Swift SDK, and cross-compile a little CLI application for Ubuntu from my Mac without having to be super clever about it (or whatever OS's we decide to produce Swift SDKs for). I'm not denying that things are probably not working right at the moment, but that's the direction we're trying to push things in.

For example, when building the compiler tools themselves in this repo, you will often see the flags -sdk / -resource-dir /home/finagolfin/build/Ninja-Release/swift-linux-aarch64/lib/swift/ used to compile against the system C/C++ sysroot and the freshly-built Swift stdlib. How do you replace that with -sdk/-sysroot? We'll probably have to add a third hidden flag -sdk-runtime that works just like -resource-dir currently does, if we ever want to remove the -resource-dir flag from building this repo itself.

In this example, -sdk points at the standard libraries (swiftCore, concurrency, etc...) and their modules. -sysroot points at the C/C++ libraries coming from the system sysroot, and -resource-dir points at the compiler resources. This will make a lot more sense once the standard library and compiler build systems are fully split from each other. The standard library build won't be installing the just-built standard library into the compiler build directory since it's a separate project, so -sdk will need to point at where the stdlib was installed to.

[finagolfin]

So I don't think getting rid of the -resource-dir is the goal of that proposal, though certainly lessening the reliance on setting it.

OK, I thought the plan was to deprecate the flag altogether, but changing the meaning of such a core compilation flag instead is going to be even harder.

Under the new definition, the -sdk flag should point to the Swift runtimes for the platform you're targeting

Just to be clear, because it is not specified in that new cross-compilation document, will the new model look directly in the specified -sdk path for the runtime modules and libraries or in -sdk/<OS> or in -sdk/usr/lib/swift/<OS>? If the last, to maintain compatibility with the old model of -sdk pointing at a C/C++ sysroot, then the new model cannot point at an arbitrary Swift resource directory anymore, such as my example above of building the compiler tools in this repo.

this will become more important as we split the SDK distribution from the per-platform toolchain monolith that we have today... This will make a lot more sense once the standard library and compiler build systems are fully split from each other.

It would be nice if whoever is planning this split would write up what files are going to go where and exactly what compiler flags will be used to access them, instead of the vague hand-waving in the current cross-compilation doc.

My eventual goal is to be able to download an Ubuntu sysroot, an Ubuntu Swift SDK, and cross-compile a little CLI application for Ubuntu from my Mac without having to be super clever about it (or whatever OS's we decide to produce Swift SDKs for).

Sounds great, but the transition process to get there looks rough.

I spent some time this weekend investigating how the limited support for an external -sdk has worked over the years, and it turns out that it only ever supported that being a C/C++ sysroot. While some limited support for it being a "full SDK," ie also containing non-Darwin Swift runtime modules and libraries in -sdk/usr/lib/swift/<OS> without also explicitly specifying that path as the -resource-dir, was added over the years to the old C++ Driver, #25990, and more recently to the new swift-driver in 6.1, swiftlang/swift-driver#1560, I found that it never really worked, because no support was ever added in the swift-frontend to look for the Swift runtime modules in -sdk/usr/lib/swift/<OS>, ie both the drivers would look in -sdk/usr/lib/swift/<OS> for a few files like swiftrt.o or libswiftCore.so but the frontend then ignores the -sdk and gets the runtime modules like Swift.swiftmodule from outside the -sdk! I'm trying to finally fix this in #79621.

Supporting such full SDKs is not too important, but I bring it up for two reasons. One is that the plan appears to be in the new cross-compilation model to look for the Swift resource directory in -sdk/usr/lib/swift, as implemented for non-Darwin Unix in swiftlang/swift-driver#1560 for swiftrt.o alone, but that has never worked for such full SDKs in the old model. The other is that by implementing that new model for just that one Unix object file, the old -sdk/-resource-dir model broke in many scenarios, ie when -sdk does not contain a Swift resource directory or just an incompatible one. I have to work around this on my Android CI ever since that swift-driver pull was merged against my objections.

I suggest you take a look at #79621 and the associated swiftlang/swift-driver#1822 and see if that fits your plans for this new cross-compilation model. Both are tiny pulls that keep both old and new cross-compilation models working, while moving the new one laid out in the cross-compilation doc forward.

[finagolfin]

OK, I spent some time diving into this and the fundamental issue appears to be on macOS, with implications for these other platforms. If a -resource-dir is set or the default resource directory next to the compiler is used, the frontend and driver set an internal variable RuntimeResourcePath and look in that path for the Swift runtime libraries and modules and so on. There is a bunch of logic in the compiler keyed to this RuntimeResourcePath variable, and it works well.

However, for the case where a full SDK is passed in, ie only the -sdk flag because it contains both C/C++ headers for a platform and Swift runtime modules/libraries, a choice was made six years ago not to set RuntimeResourcePath to <sdk>/usr/lib/swift in that full SDK, but to instead special case it and add eg <sdk>/usr/lib/swift to RuntimeLibraryImportPaths separately. This means that everywhere that compiler logic is keyed to RuntimeResourcePath, we have to duplicate it again if !SearchPathOpts.getSDKPath().empty(), as originally implemented for Swift runtime modules in #23175. This duplication has not been done everywhere, therefore full SDKs where you simply specify an -sdk path alone don't work.

However, the new cross-compilation model depends on that full SDK logic being implemented well and being able to stop using the -resource-dir flag. Currently, neither appears to be possible, at least not without sticking in a bunch of other flags as hacks that then break in all kinds of weird ways.

Based on my experiments in #79621 with setting RuntimeResourcePath also to <sdk>/usr/lib/swift, I'm guessing this was done because of some layering issues with the Darwin SDK, but linux and Windows build the Swift stdlib fine with that alternate config.

So the question is now what we do about this, I see four alternatives:

1. Get Darwin platforms working too when RuntimeResourcePath is set to <sdk>/usr/lib/swift.
2. Only set RuntimeResourcePath to <sdk>/usr/lib/swift for non-Darwin platforms, and have different cross-compilation rules for Darwin and non-Darwin.
3. Duplicate all RuntimeResourcePath logic a second time in the compiler with <SDKPath>/usr/lib/swift.
4. Abandon this plan to get rid of the -resource-dir flag for cross-compilation and keep it as a mandatory flag, as it is now.

@compnerd and @etcwilde, I don't know if you two have looked at all this relevant internal logic and if you agree with my appraisal of the current situation: let me know if you think I got anything wrong and how you want to move forward.

@beccadax, as the author of #23175, perhaps you can fill us in on why it was implemented that way and the viability of switching Darwin too to option 1. above.