[wasi][coreclr] Library tests crash with "call stack exhausted"

[lewing]

Description

On CoreCLR WASI, some library test suites crash with wasm trap: call stack exhausted:

• System.Threading.Tasks.Dataflow.Tests: the trap occurs in a long chain of task continuations (Task.NotifyDebuggerOfWaitCompletionIfNecessary ← ContinuationResultTaskFromTask<__Canon>.InnerInvoke ← WasmVirtualDispatchThunk ← ExecutionContext.RunInternal ← Task.ExecuteWithThreadLocal ...).
• System.Text.RegularExpressions.Unit.Tests: deep recursion in RegexNode.FindAndMakeLoopsAtomic (→ RegexNode.ChildCount → CastHelpers.IsInstanceOfAny → StaticsHelpers.GetGCStaticBase).

Both run the same code paths on other platforms, including browser, without exhausting the stack. On WASI the wasm call stack available to CoreCLR, and the per-frame cost of R2R code and the interpreter transition thunks, may leave less headroom. Check:

• the configured stack size for the wasihost corerun link and for wasmtime (-W max-wasm-stack);
• whether RuntimeHelpers.EnsureSufficientExecutionStack/TryEnsureSufficientExecutionStack, which the regex code uses to avoid deep recursion, reports the real remaining stack on WASI.

Seen in trimmed ReadyToRun runs of the CoreCLR WASI library tests for #134813.

Note

This issue was drafted with the help of GitHub Copilot.

[dotnet-policy-service]

Tagging subscribers to 'arch-wasm': @lewing, @pavelsavara
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @agocke
See info in area-owners.md if you want to be subscribed.