A note about Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

In https://github.com/trailbaseio/trailbase/blame/main/Cargo.toml#L40 I found an interesting comment about LTO + PGO. This bug is already fixed in the Rustc upstream: rust-lang/rust#115344 .So since Rust 1.87 (AFAIK) you can use LTO with PGO. I think it could be interesting to give PGO one more try with Trailbase.

Additionally, I propose to test Post-Link Optimization (PLO) on the project with tools like LLVM BOLT - it can bring performance improvements even after applying PGO. However, I recommend starting with PGO since PGO is more stable technology right now.

Thank you.

[ignatz]

Thanks for the update that's awesome. I'll take another look (FWIW, TB releases are still being built with 1.86, since I had issues with 1.87 for mac arm (funnily x86 was fine) and windows builds) 🙏

[ignatz]

A "quick" test and painfully slow rebuilds later, I can report that I was able to build an instrumented binary with fat LTO, collect some profiles, ... but when building an optimized binary I'm still seeing linker issues:

error: linking with `cc` failed: exit status: 12: trail(bin)
  |
  = note:  "cc" "-m64" "/tmp/rustcMQBukm/symbols.o" "<1 object files omitted>" "-Wl,--as-needed" "-Wl,-Bstatic" "/tmp/rustcMQBukm/{liblibmimalloc_sys-a78b5e35f200a864.rlib,libv8-65eb8799280e0afe.rlib,libpsm-42bc30a2f6bbf733.rlib,libtrailbase_sqlean-3e3cd623616a70a4.rlib,libsqlite_vec-64928bfad932b03c.rlib,liblibsqlite3_sys-e010cd521a2781fd.rlib,libring-5f8a0b252c0f4ca7.rlib}.rlib" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib/{libcompiler_builtins-*}.rlib" "-Wl,-Bdynamic" "-lc" "-lsqlite3" "-lssl" "-lcrypto" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-L" "/tmp/rustcMQBukm/raw-dylibs" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/libmimalloc-sys-aa9255c953e9ff80/out" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/ring-900b11d463b0235d/out" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/psm-4544cfc3dcb204b5/out" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/libsqlite3-sys-68f39ae6aaf94f79/out" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/sqlite-vec-14763278cdf37d47/out" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/build/trailbase-sqlean-7b738f620c58d3b6/out" "-L" "./bundled/sqlean/src/define" "-L" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/gn_out/obj" "-L" "<sysroot>/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/deps/trail-9c600b48a0d17e17" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-Wl,--strip-debug" "-nodefaultlibs"
  = note: some arguments are omitted. use `--verbose` to show all linker arguments
  = note: /usr/bin/ld: /home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/deps/trail-9c600b48a0d17e17.trail.ae394532ebdab341-cgu.0.rcgu.o: warning: relocation against `_ZN4ring6pbkdf216PBKDF2_HMAC_SHA117ha1ff148aef70ce5dE' in read-only section `.text._ZN5tokio7runtime4task3raw4poll17h25e9804c635fc44bE'
          /usr/bin/ld: /home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/deps/trail-9c600b48a0d17e17.trail.ae394532ebdab341-cgu.0.rcgu.o: in function `tokio::runtime::task::raw::poll':
          trail.ae394532ebdab341-cgu.0:(.text._ZN5tokio7runtime4task3raw4poll17h5de921da1f5e8fa0E+0x4cc): undefined reference to `ring::pbkdf2::PBKDF2_HMAC_SHA1'
          /usr/bin/ld: trail.ae394532ebdab341-cgu.0:(.text._ZN5tokio7runtime4task3raw4poll17h5de921da1f5e8fa0E+0x83d): undefined reference to `ring::pbkdf2::PBKDF2_HMAC_SHA1'
          /usr/bin/ld: trail.ae394532ebdab341-cgu.0:(.text._ZN5tokio7runtime4task3raw4poll17h5de921da1f5e8fa0E+0x854): undefined reference to `ring::pbkdf2::PBKDF2_HMAC_SHA1'
          /usr/bin/ld: /home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/deps/trail-9c600b48a0d17e17.trail.ae394532ebdab341-cgu.0.rcgu.o: in function `tokio::runtime::task::raw::poll':
          trail.ae394532ebdab341-cgu.0:(.text._ZN5tokio7runtime4task3raw4poll17h5d3ddc647b39393dE+0x860): undefined reference to `ring::pbkdf2::PBKDF2_HMAC_SHA1'
          /usr/bin/ld: /home/sebastian/projects/trailbase/target/x86_64-unknown-linux-gnu/release/deps/trail-9c600b48a0d17e17.trail.ae394532ebdab341-cgu.0.rcgu.o: in function `tokio::runtime::task::raw::poll':
          trail.ae394532ebdab341-cgu.0:(.text._ZN5tokio7runtime4task3raw4poll17h25e9804c635fc44bE+0x6b3): undefined reference to `ring::pbkdf2::PBKDF2_HMAC_SHA1'
          /usr/bin/ld: warning: creating DT_TEXTREL in a PIE
          cc: error: linker command failed with exit code 1 (use -v to see invocation)

  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#rustc-link-lib)

error: could not compile `trailbase-cli` (bin "trail") due to 1 previous error; 2303 warnings emitted
PGO optimized build has failed.
Cargo finished with an error (101)

[ignatz]

Update: I was able to build an optimized build with "thin" LTO (all feels still more brittle than it should). Some more testing will be needed to establish whether (no-PGO and fat LTO) or (PGO with thin LTO) is faster.

[zamazan4ik]

Sounds strange. I think it's worth to report it somewhere. Maybe at least ask in the Rustc Discord about that. Because FatLTO definitely should work fine with PGO.

[ignatz]

A few more painfully slow release builds later, the linker issue seems to be a v1.87 issue (that I briefly mentioned earlier). I'm getting the same linker error w/o PGO. So hopefully things will work themselves out over time..

The good news: PGO with thin LTO manages to speed up the benchmarks by ~20%. That's wonderful.

That said, it should probably be understood as an upper bound since I'm using the exact same workflow for profile collection and subsequent measurement. Presumably, some of this guidance will carry over to more varied workloads. Ultimately, users will always achieve the biggest gains if they collect their own profiles for their own workloads.

[ignatz]

I found briansmith/ring#2530, which suggests that the linker issues should be worked out in the next ring crate release 🤞