[SR-730] Flag to statically link Swift standard library

[drewcrawford]

Previous ID	SR-730
Radar	None
Original Reporter	@drewcrawford
Type	Bug
Status	Closed
Resolution	Done

Attachment: Download

Environment

swift-DEVELOPMENT-SNAPSHOT-2016-02-08-a

Additional Detail from JIRA

Votes	6
Component/s	Compiler
Labels	Bug, Driver
Assignee	@drewcrawford
Priority	Medium

md5: 750294aa83c3f055bad270719f53d070

relates to:

• SR-648 swift package manager should have the option to produce statically linked binaries
• SR-2280 swiftc -static-stdlib option fails on Ubuntu 14.04 & 15.10
• SR-727 (emscripten support) Compile Swift code with statically linked stdlib
• SR-9821 Swift Static Standard Library is being treated as private api

Issue Description:

If I build an executable with swiftc:

$ swiftc test.swift

Then I find out the Swift standard library was dynamically linked:

$ otool -L test
@rpath/libswiftCore.dylib (compatibility version 0.0.0, current version 0.0.0)

However, the binary distributions also have a static version that I didn't use:

/Library/Developer/Toolchains/swift-latest.xctoolchain//usr/lib/swift_static/macosx/libswiftCore.a

I want to pass e.g. -static-stdlib to swiftc to compile a program that is statically linked with the Swift standard library.

[drewcrawford]

I attempted to go diving to find the correct flags, but have not worked it out just yet.

swiftc -o foo foo.swift /usr/local/lib/swift_static/linux/libswiftCore.a /usr/local/lib/swift_static/linux/libswiftGlibc.a -licuuc -licui18n -lbsd

"works", but still generates a dynamically-linked executable.

Diving deeper, on Linux x64, somebody is passing -lswiftCore to clang++:

/usr/bin/clang++ /tmp/foo-4c5e3f.o /usr/local/lib/swift_static/linux/libswiftCore.a /usr/local/lib/swift_static/linux/libswiftGlibc.a -licuuc -licui18n -lbsd -L /usr/local/lib/swift/linux --target=x86_64-unknown-linux-gnu -Xlinker -rpath -Xlinker /usr/local/lib/swift/linux -lswiftCore @/tmp/libswiftCore-0309d2.autolink -Xlinker -T /usr/local/lib/swift/linux/x86_64/swift.ld -o foo

But I can't figure out who did that and where I can disable it.

[drewcrawford]

On OSX, the flags I'm looking for are

/Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftFoundation.a /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftObjectiveC.a /Library/Developer/Toolchains/swift-latest.xctoolchain/usr/lib/swift_static/macosx/libswiftCore.a -lc++ -framework Foundation

Which works ✅

It seems that OSX uses ld for linking, which (for reasons unknown to me) do not try to dynamically link libswiftCore.

However, Linux uses clang++ as its linker, and attempts to link libswiftCore.so no matter what I do. So I still can't figure out how to do this on Linux.

[drewcrawford]

cc: @belkadan you're the culprit here 🙂

In ToolChains.cpp appears the code

// Always add the stdlib
1175      Arguments.push_back("-lswiftCore");

We should not "always" add -lswiftCore because then the stdlib cannot be statically linked.

[belkadan]

Um. We should certainly not require people to explicitly specify -lswiftCore normally, though. What do you recommend?

[drewcrawford]

Of course. What I would recommend would be one or both of:

1. -no-stdlib, which would not use -lswiftCore and allow me to cobble together the behavior I want
2. -static-stdlib, which would just go ahead and link the standard library/runtime statically

For background, the standard library breaks frequently (e.g.,each time the last few snapshots) right now on Linux. I currently have to coordinate binary releases around language snapshots (user needs to run same snapshot as binary was built with) and I don't have good visibility into when those come out to plan my releases around it.

One or both of these would let me (with a little effort) create static binaries, so that I'm not chasing the wind every few days.

[drewcrawford]

Just a friendly bump on this. I don't see that a solution has been blessed by core.

cc: @mxcl also expressed an interest in this bug.

[drewcrawford]

Friendly bump.

At the moment I have to recompile all binaries with each new snapshot; that's quite an effort.

[ephemer]

Just so I understand- you were able to build a binary on OS X compiled from swift sources that is entirely statically linked? This would be big news for emscripten support.

Is it really just commenting out Arguments.push_back("- lswiftCore") and running build-script with --extra-swift-flags --build-static-stdlib=1? Would you might putting together a short rundown on what changes you had to make to support this?

[drewcrawford]

Sure, let me tell The Story So Far™.

You can actually build a statically-linked binary on OSX right now. This has always worked, because Xcode does it by default for OSX executables. What you must do is simply

swiftc -L path/to/swift_static -lc++ -framework Foundation -Xlinker -force_load_swift_libs

I discovered this by simply looking through Xcode build logs.

On Linux, however, it doesn't work, for starters, because Linux linker doesn't have -force_load_swift_libs. I believe the correct incantation is

swiftc -licuuc -licu18n -lbsd /path/to/swift_static/a.a /path/to/swift_static/b.a /path/to/swift_static/c.a ...

However because swiftc forcibly injects -lswiftCore on Linux this results in a library that is both statically and dynamically linked. Which defeats the purpose.

I was hoping to hear back from @belkadan or perhaps even @mxcl regarding how to proceed on the -lswiftCore issue (we need an option to disable it I guess?), but I don't yet know what is acceptable on that question.

[mxcl]

We should have this feature and then open a bug for the swift Linux driver to fix it.

I want to propose this goes into some other file than Package.swift.

My rationale is that the Package.swift describes modules and products, but not the installation or system linkage of those products.

The enduser should have complete control over the final binary composition of their products, and not be at the mercy of their dependencies.

[drewcrawford]

This bug is the swift driver bug.

I'm not especially concerned with how this gets exposed from SwiftPM other than it should, but to me that is kind of a side issue. Right now we can't even statically link a Swift binary with a makefile.

[mxcl]

My bad, got mixed up since there is another bug open on SwiftPM for this.

[drewcrawford]

So I took a pass at this, you can see the WIP at https://github.com/drewcrawford/swift/tree/static-stdlib

That tree introduces a new flag, -static-stdlib, that in theory statically links the stdlib. This is Linux-only, I haven't yet gotten to OSX.

The immediate problem is that something odd is happening with the binaries that get built. I try to build this simple hello world program:

print("Hello world")

When linked dynamically against libswiftCore.so, it works fine. However, when linked statically against libswiftCore.a, the hello world program prints this mysterious thing:

String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none))

It looks like the internal representation of a string, but I'm not sure why this happens. Can someone ID this print and figure out why I am printing it?

Here's a verbose build, and these invocations can reproduce the problem without any stuff in my tree (which is merely driver stuff):

'/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swiftc' -v  -static-stdlib test2.swift
Swift version 3.0-dev (LLVM 699a786c15, Clang 77080f2c03, Swift 21c9372d82)
Target: x86_64-unknown-linux-gnu
/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift -frontend -c -primary-file test2.swift -target x86_64-unknown-linux-gnu -disable-objc-interop -color-diagnostics -module-name test2 -o /tmp/test2-9cbb9a.o
/swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/bin/swift-autolink-extract /tmp/test2-9cbb9a.o -o /tmp/test2-49baa1.autolink
/usr/bin/clang++ -ldl -lpthread -lbsd -licui18n -licuuc /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_begin.o /tmp/test2-9cbb9a.o -L /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux --target=x86_64-unknown-linux-gnu -Xlinker -rpath -Xlinker /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift_static/linux -lswiftCore @/tmp/test2-49baa1.autolink /swift-dev/build/Ninja-ReleaseAssert/swift-linux-x86_64/lib/swift/linux/x86_64/swift_end.o -o test2

[ephemer]

Great stuff for working on this!

Off the top of my head, this is the same thing that was happening before either
a) the swift.ld linker script, or
b) the runtime protocol conformance mapping
... was incorporated into the Android port of Swift. I forget which, my recollection is leaning me towards the runtime. I think it was something to do with the type introspection.

[ephemer]

I'm not at my computer so I can't edit my comment above.. Try declaring an enum or conforming a custom Struct or Class to a protocol, if the program crashes on startup it's the runtime.

The swift.ld linker script was changed (by an apple employee from memory) to an inline assembly version as far as I understand, I lost track of that somewhere along the way. I understand the script basically performed a recursive loop to find the "end-of-chain" memory space for things like printing values (instead of just printing info about the Struct wrapper, as you're seeing here). So this contradicts what I said above, I now actually think it's more likely to be swift.ld, assuming it really is one of the two issues. In any case it's worth checking whether the linker script (in whatever current form it has) is being invoked in your statically linked branch