Move CoreCLR over to the managed wait subsystem #117788

jkoritzinsky · 2025-07-17T21:36:17Z

Use the shared managed wait subsystem for CoreCLR's managed code instead of the Win32 PAL

Also, remove the named mutex support from the CoreCLR PAL as well as Mutex support.

Unblocks #115685

src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Unix.cs

src/coreclr/vm/threads.cpp

src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs

src/libraries/Common/src/Interop/Unix/System.Native/Interop.GetPageSize.cs

src/coreclr/vm/threads.cpp

…y/runtime into coreclr-managed-wait

jkoritzinsky · 2025-12-09T23:22:44Z

Looks like I need to do some perf work in the wait subsystem. Time to first request is much worse and requests/sec regresses as well.

JSON benchmark results

| application | PR | Main | | --------------------------------------- | --------------------------------- | --------------------------------- | | Max Process CPU Usage (%) | 90 | 90 | | Max Cores usage (%) | 5,030 | 5,036 | | Max Working Set (MB) | 115 | 119 | | Max Private Memory (MB) | 894 | 897 | | Build Time (ms) | 8,267 | 9,119 | | Start Time (ms) | 470 | 280 | | Published Size (KB) | 109,039 | 109,039 | | Symbols Size (KB) | 34 | 34 | | .NET Core SDK Version | 10.0.100-rtm.25513.102 | 10.0.100-rtm.25513.102 | | ASP.NET Core Version | 10.0.0-rtm.25513.102+b502b6eeec0d | 10.0.0-rtm.25513.102+b502b6eeec0d | | .NET Runtime Version | 10.0.0-rtm.25513.102+b502b6eeec0d | 10.0.0-rtm.25513.102+b502b6eeec0d | | Max Global CPU Usage (%) | 90 | 92 | | Max CPU Usage (%) | 89 | 90 | | Max Working Set (MB) | 121 | 123 | | Max GC Heap Size (MB) | 19 | 18 | | Size of committed memory by the GC (MB) | 27 | 31 | | Max Number of Gen 0 GCs / sec | 32.00 | 29.00 | | Max Number of Gen 1 GCs / sec | 1.00 | 2.00 | | Max Number of Gen 2 GCs / sec | 1.00 | 1.00 | | Max Gen 0 GC Budget (MB) | 33 | 15 | | Max Time in GC (%) | 17.00 | 2.00 | | Max Gen 0 Size (B) | 560 | 1,248 | | Max Gen 1 Size (B) | 2,608,768 | 963,760 | | Max Gen 2 Size (B) | 5,856,456 | 5,710,120 | | Max LOH Size (B) | 88,680 | 88,680 | | Max POH Size (B) | 1,606,608 | 1,594,248 | | Max Allocation Rate (B/sec) | 275,062,672 | 277,456,832 | | Max GC Heap Fragmentation (%) | 1,352% | 340% | | # of Assemblies Loaded | 107 | 101 | | Max Exceptions (#/s) | 458 | 442 | | Max Lock Contention (#/s) | 973 | 1,022 | | Max ThreadPool Threads Count | 76 | 76 | | Max ThreadPool Queue Length | 104 | 68 | | Max ThreadPool Items (#/s) | 2,574,119 | 2,593,353 | | Max Active Timers | 1 | 1 | | IL Jitted (B) | 541,472 | 367,434 | | Methods Jitted | 7,263 | 3,974 | | Load Working Set - P90 (MB) | 115 | 118 | | Load CPU Usage - P90 (%) | 85 | 88 |

load	PR	Main
Max Process CPU Usage (%)	37	37
Max Cores usage (%)	2,058	2,087
Max Working Set (MB)	44	45
Max Private Memory (MB)	358	358
Build Time (ms)	6,782	7,261
Start Time (ms)	0	0
Published Size (KB)	72,280	72,280
Symbols Size (KB)	0	0
.NET Core SDK Version	8.0.416	8.0.416
ASP.NET Core Version	8.0.22+ee4174799332	8.0.22+ee4174799332
.NET Runtime Version	8.0.22+a2266c728f63	8.0.22+a2266c728f63
Max Global CPU Usage (%)	56	55
First Request (ms)	422	279
Requests/sec	1,699,566	1,708,722
Requests	25,663,553	25,800,351
Mean latency (ms)	0.16	0.16
Max latency (ms)	6.78	6.91
Bad responses	0	0
Socket errors	0	0
Read throughput (MB/s)	301.47	303.10
Latency 50th (ms)	0.13	0.13
Latency 75th (ms)	0.18	0.18
Latency 90th (ms)	0.25	0.25
Latency 99th (ms)	0.81	0.81

Plaintext benchmark results

application	PR	Main
Max Process CPU Usage (%)	99	96
Max Cores usage (%)	5,528	5,393
Max Working Set (MB)	97	102
Max Private Memory (MB)	847	847
Build Time (ms)	9,108	8,750
Start Time (ms)	468	276
Published Size (KB)	109,039	109,039
Symbols Size (KB)	34	34
.NET Core SDK Version	10.0.100-rtm.25513.102	10.0.100-rtm.25513.102
ASP.NET Core Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
.NET Runtime Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
Max Global CPU Usage (%)	100	98
Max CPU Usage (%)	99	96
Max Working Set (MB)	101	106
Max GC Heap Size (MB)	10	10
Size of committed memory by the GC (MB)	8	8
Max Number of Gen 0 GCs / sec	2.00	2.00
Max Number of Gen 1 GCs / sec	1.00	1.00
Max Number of Gen 2 GCs / sec	0.00	0.00
Max Gen 0 GC Budget (MB)	13	13
Max Time in GC (%)	17.00	12.00
Max Gen 0 Size (B)	0	0
Max Gen 1 Size (B)	2,621,680	2,621,680
Max Gen 2 Size (B)	1,100,224	1,145,040
Max LOH Size (B)	88,680	88,680
Max POH Size (B)	1,639,568	1,672,528
Max Allocation Rate (B/sec)	5,522,280	5,932,352
Max GC Heap Fragmentation (%)	953%	863%
# of Assemblies Loaded	107	101
Max Exceptions (#/s)	472	454
Max Lock Contention (#/s)	10,645	11,487
Max ThreadPool Threads Count	76	76
Max ThreadPool Queue Length	188	172
Max ThreadPool Items (#/s)	742,071	745,110
Max Active Timers	1	1
IL Jitted (B)	529,015	358,353
Methods Jitted	7,180	3,913
Load Working Set - P90 (MB)	97	102
Load CPU Usage - P90 (%)	91	94

load	PR	Main
Max Process CPU Usage (%)	29	27
Max Cores usage (%)	1,597	1,539
Max Working Set (MB)	43	42
Max Private Memory (MB)	360	360
Build Time (ms)	7,326	6,503
Start Time (ms)	1	1
Published Size (KB)	72,280	72,280
Symbols Size (KB)	0	0
.NET Core SDK Version	8.0.416	8.0.416
ASP.NET Core Version	8.0.22+ee4174799332	8.0.22+ee4174799332
.NET Runtime Version	8.0.22+a2266c728f63	8.0.22+a2266c728f63
Max Global CPU Usage (%)	65	65
First Request (ms)	410	269
Requests/sec	8,294,536	8,490,340
Requests	125,248,638	128,203,039
Mean latency (ms)	0.36	0.35
Max latency (ms)	11.09	11.75
Bad responses	0	0
Socket errors	0	0
Read throughput (MB/s)	1,300.48	1,331.20
Latency 50th (ms)	0.34	0.32
Latency 75th (ms)	0.60	0.56
Latency 90th (ms)	2.26	1.44
Latency 99th (ms)	0.00	0.00

VSadov · 2025-12-09T23:52:34Z

Time to first request is much worse

I see the Methods Jitted almost doubled. If that is included in the time, it might be one of the reasons.
Something was not crossgenned?

…ith a comment stating that we know that this can cause deadlocks for users that write bad code.

jkoritzinsky · 2025-12-10T00:59:10Z

I'm rerunning now to grab a trace and see if I can figure out why methods jitted is so high.

Might just be R2R major version differences between the aspnetcore version I used (pulled from .NET 10) and the .NET runtime version (where we've already bumped the min R2R major version twice) causing all of the R2R code for aspnetcore to be thrown out.

sebastienros · 2025-12-10T01:15:27Z

For this reason we usually suggest to benchmark the two locally built versions. You only have to build baseline once. Apples to apples.

jkoritzinsky · 2025-12-10T01:25:11Z

That's a tomorrow problem :)

…ed to not protect against

jkoritzinsky · 2025-12-10T19:34:59Z

Here's the results after doing an apples-to-apples comparison (Main is locally built from the last merge point into the PR branch)

JSON benchmark results

application	PR	Main
Max Process CPU Usage (%)	90	90
Max Cores usage (%)	5,030	5,015
Max Working Set (MB)	115	112
Max Private Memory (MB)	894	927
Build Time (ms)	8,267	9,039
Start Time (ms)	470	472
Published Size (KB)	109,039	109,039
Symbols Size (KB)	34	34
.NET Core SDK Version	10.0.100-rtm.25513.102	10.0.100-rtm.25513.102
ASP.NET Core Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
.NET Runtime Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
Max Global CPU Usage (%)	90	92
Max CPU Usage (%)	89	90
Max Working Set (MB)	121	121
Max GC Heap Size (MB)	19	17
Size of committed memory by the GC (MB)	27	30
Max Number of Gen 0 GCs / sec	32.00	28.00
Max Number of Gen 1 GCs / sec	1.00	1.00
Max Number of Gen 2 GCs / sec	1.00	1.00
Max Gen 0 GC Budget (MB)	33	19
Max Time in GC (%)	17.00	6.00
Max Gen 0 Size (B)	560	126,688
Max Gen 1 Size (B)	2,608,768	5,333,680
Max Gen 2 Size (B)	5,856,456	1,099,664
Max LOH Size (B)	88,680	88,680
Max POH Size (B)	1,606,608	1,577,768
Max Allocation Rate (B/sec)	275,062,672	276,387,088
Max GC Heap Fragmentation (%)	1,352%	3,580%
# of Assemblies Loaded	107	107
Max Exceptions (#/s)	458	450
Max Lock Contention (#/s)	973	1,118
Max ThreadPool Threads Count	76	76
Max ThreadPool Queue Length	104	100
Max ThreadPool Items (#/s)	2,574,119	2,637,707
Max Active Timers	1	1
IL Jitted (B)	541,472	538,849
Methods Jitted	7,263	7,229
Load Working Set - P90 (MB)	115	108
Load CPU Usage - P90 (%)	85	87

load	PR	Main
Max Process CPU Usage (%)	37	37
Max Cores usage (%)	2,058	2,045
Max Working Set (MB)	44	44
Max Private Memory (MB)	358	358
Build Time (ms)	6,782	6,964
Start Time (ms)	0	0
Published Size (KB)	72,280	72,280
Symbols Size (KB)	0	0
.NET Core SDK Version	8.0.416	8.0.416
ASP.NET Core Version	8.0.22+ee4174799332	8.0.22+ee4174799332
.NET Runtime Version	8.0.22+a2266c728f63	8.0.22+a2266c728f63
Max Global CPU Usage (%)	56	55
First Request (ms)	422	421
Requests/sec	1,699,566	1,697,701
Requests	25,663,553	25,633,992
Mean latency (ms)	0.16	0.16
Max latency (ms)	6.78	5.82
Bad responses	0	0
Socket errors	0	0
Read throughput (MB/s)	301.47	301.14
Latency 50th (ms)	0.13	0.13
Latency 75th (ms)	0.18	0.18
Latency 90th (ms)	0.25	0.25
Latency 99th (ms)	0.81	0.85

Plaintext benchmark results

application	PR	Main
Max Process CPU Usage (%)	99	100
Max Cores usage (%)	5,528	5,573
Max Working Set (MB)	97	98
Max Private Memory (MB)	847	847
Build Time (ms)	9,108	9,189
Start Time (ms)	468	469
Published Size (KB)	109,039	109,039
Symbols Size (KB)	34	34
.NET Core SDK Version	10.0.100-rtm.25513.102	10.0.100-rtm.25513.102
ASP.NET Core Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
.NET Runtime Version	10.0.0-rtm.25513.102+b502b6eeec0d	10.0.0-rtm.25513.102+b502b6eeec0d
Max Global CPU Usage (%)	100	100
Max CPU Usage (%)	99	99
Max Working Set (MB)	101	102
Max GC Heap Size (MB)	10	10
Size of committed memory by the GC (MB)	8	8
Max Number of Gen 0 GCs / sec	2.00	2.00
Max Number of Gen 1 GCs / sec	1.00	1.00
Max Number of Gen 2 GCs / sec	0.00	0.00
Max Gen 0 GC Budget (MB)	13	13
Max Time in GC (%)	17.00	5.00
Max Gen 0 Size (B)	0	0
Max Gen 1 Size (B)	2,621,680	2,621,568
Max Gen 2 Size (B)	1,100,224	1,092,184
Max LOH Size (B)	88,680	88,680
Max POH Size (B)	1,639,568	1,598,368
Max Allocation Rate (B/sec)	5,522,280	5,859,776
Max GC Heap Fragmentation (%)	953%	1,177%
# of Assemblies Loaded	107	107
Max Exceptions (#/s)	472	466
Max Lock Contention (#/s)	10,645	11,844
Max ThreadPool Threads Count	76	76
Max ThreadPool Queue Length	188	135
Max ThreadPool Items (#/s)	742,071	758,176
Max Active Timers	1	1
IL Jitted (B)	529,015	526,490
Methods Jitted	7,180	7,151
Load Working Set - P90 (MB)	97	98
Load CPU Usage - P90 (%)	91	95

load	PR	Main
Max Process CPU Usage (%)	29	27
Max Cores usage (%)	1,597	1,499
Max Working Set (MB)	43	43
Max Private Memory (MB)	360	360
Build Time (ms)	7,326	7,535
Start Time (ms)	1	1
Published Size (KB)	72,280	72,280
Symbols Size (KB)	0	0
.NET Core SDK Version	8.0.416	8.0.416
ASP.NET Core Version	8.0.22+ee4174799332	8.0.22+ee4174799332
.NET Runtime Version	8.0.22+a2266c728f63	8.0.22+a2266c728f63
Max Global CPU Usage (%)	65	65
First Request (ms)	410	422
Requests/sec	8,294,536	8,280,244
Requests	125,248,638	125,029,739
Mean latency (ms)	0.36	0.35
Max latency (ms)	11.09	9.61
Bad responses	0	0
Socket errors	0	0
Read throughput (MB/s)	1,300.48	1,300.48
Latency 50th (ms)	0.34	0.32
Latency 75th (ms)	0.60	0.56
Latency 90th (ms)	2.26	1.14
Latency 99th (ms)	0.00	0.00

With an apples-to-apples comparison, the difference in First Request (ms) goes away (lending credit to the R2R rejection hypothesis) and the PR actually has an improved Requests/sec instead of a slight regression!

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

jkotas

Thank you! Great work!

jkoritzinsky · 2025-12-11T12:06:36Z

/ba-g WBT failures and a spurious infra problem

jkoritzinsky added area-System.Threading runtime-coreclr specific to the CoreCLR runtime labels Jul 17, 2025

dotnet-policy-service bot assigned jkoritzinsky Jul 17, 2025

jkoritzinsky mentioned this pull request Jul 17, 2025

[Android] Add support for named mutex on mobile platforms in CoreCLR #117760

Open

jkoritzinsky added this to the 11.0.0 milestone Jul 17, 2025

jkoritzinsky added the blocked Issue/PR is blocked on something - see comments label Jul 17, 2025

jkoritzinsky force-pushed the coreclr-managed-wait branch from 25446d0 to 63dc9af Compare July 21, 2025 19:44

jkoritzinsky force-pushed the coreclr-managed-wait branch 3 times, most recently from 06de3b2 to 63a1aa6 Compare July 31, 2025 16:44

This was referenced Jul 31, 2025

slow macOS - "##[error]The job running on agent Azure Pipelines 9 ran longer than the maximum time of 60 minutes." dotnet/dnceng#1883

Open

[browser] Wasm.Build.Tests timeout - Timed out after 10s waiting for 'WASM EXIT' #116697

Open

jkoritzinsky mentioned this pull request Aug 5, 2025

Move the implementation of Monitor to managed in CoreCLR #118371

Merged

teo-tsirpanis reviewed Aug 5, 2025

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Threading/LowLevelLifoSemaphore.Unix.cs Outdated Show resolved Hide resolved

jkotas reviewed Aug 7, 2025

View reviewed changes

src/coreclr/vm/threads.cpp Outdated Show resolved Hide resolved

jkoritzinsky force-pushed the coreclr-managed-wait branch from 63a1aa6 to 2b610bf Compare August 14, 2025 18:54

jkotas reviewed Aug 14, 2025

View reviewed changes

src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs Outdated Show resolved Hide resolved

jkotas reviewed Aug 14, 2025

View reviewed changes

src/libraries/Common/src/Interop/Unix/System.Native/Interop.GetPageSize.cs Outdated Show resolved Hide resolved

jkoritzinsky force-pushed the coreclr-managed-wait branch from 4a27a27 to ac08ce6 Compare August 14, 2025 23:13

jkotas reviewed Aug 14, 2025

View reviewed changes

src/coreclr/vm/threads.cpp Outdated Show resolved Hide resolved

jkotas reviewed Aug 15, 2025

View reviewed changes

src/coreclr/vm/threads.cpp Outdated Show resolved Hide resolved

jkoritzinsky force-pushed the coreclr-managed-wait branch from 8838c10 to 9af82b0 Compare August 21, 2025 16:47

jkoritzinsky added 2 commits December 9, 2025 22:36

KeepAlive delegates in new test

f31d8da

Merge branch 'coreclr-managed-wait' of https://github.com/jkoritzinsk…

36f772c

…y/runtime into coreclr-managed-wait

Remove the detached cleanup thread and just do it on the finalizer, w…

3c24106

…ith a comment stating that we know that this can cause deadlocks for users that write bad code.

Remove the test that validates the finalizer thread behavior we decid…

3b3557b

…ed to not protect against

jkoritzinsky marked this pull request as ready for review December 10, 2025 19:35

jkoritzinsky requested a review from steveisok as a code owner December 10, 2025 19:35

Copilot AI review requested due to automatic review settings December 10, 2025 19:35

jkoritzinsky requested review from MichalStrehovsky, lateralusX, thaystg and vitek-karas as code owners December 10, 2025 19:35

jkoritzinsky requested a review from jkotas December 10, 2025 19:35

Copilot AI reviewed Dec 10, 2025

View reviewed changes

jkotas approved these changes Dec 10, 2025

View reviewed changes

jkoritzinsky enabled auto-merge (squash) December 10, 2025 21:21

jkoritzinsky merged commit 421445e into dotnet:main Dec 11, 2025
159 of 164 checks passed

jkoritzinsky deleted the coreclr-managed-wait branch December 11, 2025 12:06

dotnet-maestro bot mentioned this pull request Dec 13, 2025

[main] Source code updates from dotnet/runtime dotnet/dotnet#3873

Open

LoopedBard3 mentioned this pull request Dec 16, 2025

[Perf] Linux/x64: 7 Regressions on 12/11/2025 12:41:46 PM +00:00 #122592

Open

agocke mentioned this pull request Dec 18, 2025

Regression in gcstress-extra in https://github.com/dotnet/runtime/pull/117788 #122650

Closed

Move CoreCLR over to the managed wait subsystem #117788

Move CoreCLR over to the managed wait subsystem #117788

Uh oh!

Conversation

jkoritzinsky commented Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jkoritzinsky commented Dec 9, 2025

Uh oh!

VSadov commented Dec 9, 2025

Uh oh!

jkoritzinsky commented Dec 10, 2025

Uh oh!

sebastienros commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jkoritzinsky commented Dec 10, 2025

Uh oh!

jkoritzinsky commented Dec 10, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jkotas left a comment

Choose a reason for hiding this comment

Uh oh!

jkoritzinsky commented Dec 11, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jkoritzinsky commented Jul 17, 2025 •

edited

Loading

sebastienros commented Dec 10, 2025 •

edited

Loading