Regression in compile time when optimizing for size

[crawford]

After updating to Rust 1.57, I'm seeing a huge increase in compilation time for an embedded project that I'm working on. The code is unfortunately not open source, but I'm hoping I can provide enough information here to work around that. If I've forgotten to include anything or made bad assumptions, please let me know.

I don't have specific numbers for the slowdown because I haven't waiting long enough for the compilation to finish, but I can say that what used to take ~90 seconds for a full build is now taking more than 10 minutes. The project targets thumbv7em-none-eabihf and is optimized for size. Here are the profiles defined in the cargo manifest:

[profile.dev]
panic = "abort"
codegen-units = 1
debug = true
opt-level = "s"
lto = false

[profile.release]
panic = "abort"
codegen-units = 1
debug = true
lto = true
opt-level = "s"

Here's the interesting thing: we have seen no increase in compilation time when building the main binary. It's the tests that are actually exhibiting the behavior. The tests target x86_64-unknown-linux-gnu and are built with cargo test. If I remove opt-level = "s" from the profiles, I'm able to build the tests just as quickly as before.

I bisected the nightlies and determined that this slowdown seems to have been introduced into nightly-2021-10-14; nightly-2021-10-13 and 1.56.1 don't exhibit this behavior.

[camelid]

I bisected the nightlies and determined that this slowdown seems to have been introduced into nightly-2021-10-14; nightly-2021-10-13 and 1.56.1 don't exhibit this behavior.

@crawford could you use cargo-bisect-rustc to try to bisect to an exact commit? You'll probably want to use the --prompt flag.

[hkratz]

Or the new --timeout flag

    -t, --timeout <timeout>
            Assume failure after specified number of seconds (for bisecting hangs)

[crawford]

@camelid I gave that a shot last night, but I was having trouble getting it to run correctly. Your suggestion to use --prompt allowed me to figure out what was going wrong, so thank you! (I'm surprised -v doesn't print the cargo output)

@hkratz I'm on NixOS, which requires some patching of binaries before they can be run, so (long story short) I'm using an older pre-built version which doesn't have the timeout flag. I do have the timeout program though. I'm looking forward to being able to use the timeout flag directly!

Thank you for both of your suggestions. I'm really glad I was able to get this running on my system. What a fantastic tool!

Here's the output (regression in ef4b306):

********************************************************************************
Regression in ef4b3069baafa75bc047a80b58cc130e80576a4e
********************************************************************************

searched nightlies: from nightly-2021-10-13 to nightly-2021-10-14
regressed nightly: nightly-2021-10-14
searched commits: from https://github.com/rust-lang/rust/commit/d7c97a02d1215e4ef26c31cb72dbaf16fd548b2c to https://github.com/rust-lang/rust/commit/dfc5add915e8bf4accbb7cf4de00351a7c6126a1
regressed commit: https://github.com/rust-lang/rust/commit/ef4b3069baafa75bc047a80b58cc130e80576a4e

[camelid]

cc @the8472

[the8472]

No concrete idea, just stabs in the dark:

Since this is optimization-level dependent I suspect LLVM is going crazy somewhere. This should be testable with -Z time-passes or -Z self-profile

The added inline annotation itself shouldn't cause rustc to generate massive amounts of additional IR, I think. Unless maybe the code somehow contains many more &mut Iterator types than Iterator types, e.g. via &mut &mut &mut Iterator chains? This should be testable with llvm-lines

Edit: Another possibility is that your test cases contain some code triggering pathological behavior while the release build doesn't.

Here's the interesting thing: we have seen no increase in compilation time when building the main binary. It's the tests that are actually exhibiting the behavior. The tests target x86_64-unknown-linux-gnu and are built with cargo test.

The profiles differ in the lto flag. Does lto = true make a different for the tests?

Edit: Additionally they also differ in the defaulted incremental parameter, so that needs to be checked too.

If I remove opt-level = "s" from the profiles, I'm able to build the tests just as quickly as before.

That covers s and 0 (default for dev). Have you also tried other opt levels? z, 1, 2, 3?

[zirill]

I also noticed that there is a dramatic increase in time with each update.
~ 2m (upd) ~ 8m (upd) ~ 27m
What will happen next?)

+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| Item                                             | Self time | % of total time | Time     | Item count | Incremental result hashing time |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| evaluate_obligation                              | 1082.39s  | 83.132          | 1082.48s | 18959      | 3.64ms                          |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| codegen_fulfill_obligation                       | 110.69s   | 8.501           | 1179.24s | 6414       | 4.21ms                          |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| normalize_projection_ty                          | 63.77s    | 4.898           | 63.81s   | 3456       | 10.11ms                         |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_passes                                      | 17.57s    | 1.350           | 17.57s   | 1          | 0.00ns                          |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| LLVM_module_codegen_emit_obj                     | 9.86s     | 0.757           | 9.86s    | 257        | 0.00ns                          |
+--------------------------------------------------+-----------+-----------------+----------+------------+---------------------------------+
| run_linker                                       | 5.76s     | 0.443           | 5.76s    | 1          | 0.00ns                          |

[the8472]

@zirill just to check, the regression in your case is also specific to opt-level=s? Otherwise that would likely be a separate issue. evaluate_obligation would suggest it might be #20304