Enable interpreter stack overflow handling

[janvorli]

Stack overflow of the interpreter stack was not being handled yet. The stack overflow coreclr test was failing due to that.

This change adds checks when a called method frame would not fit the interpreter stack and fires the stack overflow handling in that case.

I also had to fix the Thread::UnwindToFirstManagedFrame to work for the interpreter frames. Before that, it would not detect them and it would unwind through the InterpExecMethod and further native frames.

[dotnet-policy-service]

Tagging subscribers to this area: @BrzVlad, @janvorli, @kg
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR adds stack overflow handling for the interpreter stack, which was previously not being handled, causing CoreCLR stack overflow tests to fail. The changes implement checks to detect when a called method frame would exceed the interpreter stack limits and trigger appropriate stack overflow handling.

Key Changes:

• Added HandleInterpreterStackOverflow function to properly handle stack overflow exceptions in the interpreter
• Added stack overflow checks before allocating new interpreter frames
• Fixed Thread::VirtualUnwindToFirstManagedCallFrame to properly detect and unwind interpreter frames

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
src/coreclr/vm/stackwalk.cpp	Added logic to detect interpreter frames during stack unwinding and set context appropriately when unwinding past the topmost interpreter frame
src/coreclr/vm/interpexec.cpp	Added HandleInterpreterStackOverflow function and stack overflow checks at frame entry points; reorganized IP initialization to occur before stack overflow checks

[davidwrighton]

This looks fine for StackOverflow situations caused on the interpreter stack, but what about when the thread OS stack is in trouble. Do we want to consider comparing the InterpMethodContextFrame's address to GetThread()->GetCachedStackSufficientExecutionLimit()and doing something there? We have several alloca's in the interpreter codebase, all of which are technically capable of causing overflows.

[janvorli]

This looks fine for StackOverflow situations caused on the interpreter stack, but what about when the thread OS stack is in trouble.

That already worked - it triggers real stack overflow and gets handled properly. One of the stack overflow tests hits that.