PollGC rewrite and TS_CatchAtSafePoint

[filipnavara]

PR #109378 has rewritten the JIT_PollGC code into a managed helper:

runtime/src/coreclr/System.Private.CoreLib/src/System/Threading/Thread.CoreCLR.cs

Lines 500 to 524 in 365dde3

	private static unsafe void PollGC()
	{
	NativeThreadState catchAtSafePoint = ((NativeThreadClass*)Thread.DirectOnThreadLocalData.pNativeThread)->m_State & NativeThreadState.TS_CatchAtSafePoint;
	if (catchAtSafePoint != NativeThreadState.None)
	{
	ThreadNative_PollGC();
	}
	}
	[StructLayout(LayoutKind.Sequential)]
	private struct NativeThreadClass
	{
	public NativeThreadState m_State;
	}
	private enum NativeThreadState
	{
	None = 0,
	TS_AbortRequested = 0x00000001, // Abort the thread
	TS_DebugSuspendPending = 0x00000008, // Is the debugger suspending threads?
	TS_GCOnTransitions = 0x00000010, // Force a GC on stub transitions (GCStress only)
	// We require (and assert) that the following bits are less than 0x100.
	TS_CatchAtSafePoint = (TS_AbortRequested | TS_DebugSuspendPending | TS_GCOnTransitions),
	};

The rewrite has transferred the original CatchAtSafePoint() check into managed code, but it doesn't do the same thing. The unmanaged implementation of CatchAtSafePoint checks the g_TrapReturningThreads flag:

runtime/src/coreclr/vm/threads.h

Lines 859 to 871 in 365dde3

	DWORD CatchAtSafePoint()
	{
	CONTRACTL
	{
	NOTHROW;
	GC_NOTRIGGER;
	MODE_COOPERATIVE;
	}
	CONTRACTL_END;
	return g_TrapReturningThreads & 1 ||
	HasThreadStateOpportunistic(TS_CatchAtSafePoint);
	}

Since the managed implementation doesn't do that I do not see how it will ever call the ThreadNative_PollGC P/Invoke in the usual cases because the flags are apparently not set (aside for thread aborts, GC Stress and debugger aborts).

/cc @davidwrighton

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[filipnavara]

Seems like the only reason this worked to some extent at all is that the PollGC call was hijackable through return address.