Intermittent SIGSEGV in GetThreadLocalStaticBase on macOS arm64 #127776

Activity

1. MichaelSimons commented on May 4, 2026

   

   MichaelSimons

   on May 4, 2026

   Last edited by MichaelSimons

   MemberAuthor

   More actions

   Attached crashreport.json files (contain full native + managed stacks for all threads):

   These files are too large to paste inline but can be downloaded from the Helix artifact storage:

   • build-1405265-crashreport.json (146 KB)
   • build-1404596-crashreport.json (333 KB)
   • build-1403714-crashreport.json (327 KB)

   These links may expire after ~20 days.
2. dotnet-policy-service

   added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on May 4, 2026
3. github-actions

   added

   area-System.Collections

   on May 4, 2026
4. dotnet-policy-service commented on May 4, 2026

   

   dotnet-policy-servicebot

   on May 4, 2026 – with Dotnet Policy Service

   Contributor

   More actions

   Tagging subscribers to this area: @dotnet/area-system-collections
   See info in area-owners.md if you want to be subscribed.
5. jkotas

   added

   area-VM-coreclr

   and removed

   area-System.Collections

   on May 5, 2026
6. github-project-automation

   added this to AppModelon May 5, 2026
7. jkotas commented on May 5, 2026

   

   jkotas

   on May 5, 2026

   Member

   More actions

   cc @mangod9
8. dotnet-policy-service commented on May 5, 2026

   

   dotnet-policy-servicebot

   on May 5, 2026 – with Dotnet Policy Service

   Contributor

   More actions

   Tagging subscribers to this area: @agocke
   See info in area-owners.md if you want to be subscribed.
9. mangod9 commented on May 5, 2026

   

   mangod9

   on May 5, 2026

   Member

   More actions

   Symptoms

   • Intermittent SIGSEGV in GetThreadLocalStaticBase(TLSIndex) on macOS ARM64 (Apple Silicon)
   • Occurs in Helix CI during parallel test execution
   • Crashes started appearing in recent months (after Oct 2024)
   • All crashes at the same instruction offset (+0x478): ldr x8, [x20, #0x28] with x20 = NULL

   Crash Stack Trace (all 3 dumps identical)

   __wait4_nocancel
   PROCCreateCrashDump
   PROCCreateCrashDumpIfEnabled
   invoke_previous_action                    ← signal handler
   _sigtramp
   GetThreadLocalStaticBase(TLSIndex) +0x478 ← CRASH: ldr x8, [x20, #0x28], x20=NULL
   GetThreadStaticsByMethodTable +0xc4       ← QCall from managed
     ... (managed frames) ...
   MethodTable::RunClassInitEx               ← class static constructor
   MethodTable::DoRunClassInitThrowing
   InitClassHelper
     ... (managed frames) ...
   RuntimeMethodHandle_InvokeMethod          ← reflection invoke (test harness)
   

   Dump Evidence

   Consistent Pattern Across All 3 Dumps

   Dump	Threads	Crash Thread	Concurrent Thread
   coredump.9012	26	RunClassInitEx → GetThreadStaticsByMethodTable	MethodTableBuilder::SetupMethodTable2 (allocating TLS index)
   coredump.4125	32	KickOffThread → GetThreadStaticsByMethodTable	Writer already completed
   coredump.925	26	RunClassInitEx → GetThreadStaticsByMethodTable	JITting during parallel RunClassInitEx

   Register State at Crash (dump 9012, thread 0x261ae)

   • x20 = 0x0 — pMT (MethodTable pointer) is NULL
   • x19 = 0x0e — TLSIndex: type=NonCollectible (top byte 0x00), raw index=14
   • Faulting instruction: ldr x8, [x20, #0x28] → loading pMT->m_pEEClass from NULL+0x28

   Map State (dump 9012)

   • g_pThreadStaticNonCollectibleTypeIndices → map struct at 0x600000e80280
   • pMap = 0x600002098800, m_maxIndex = 16 (minimum/initial size — no grow occurred)
   • g_NextNonCollectibleTlsSlot = 15 → index 14 was the last allocated NonCollectible index
   • Map entries at dump time:
     • Indices 0–1: 0 (reserved/unused)
     • Indices 2–14: populated with valid MethodTable pointers
     • Index 14 = 0x10b767f31 (valid at dump time, but was NULL at crash time)
     • Index 15: 0 (not yet allocated)
   • Crashing thread's ThreadLocalData.cNonCollectibleTlsData = 24 (TLS array exists, slot 14 was null → first access on this thread)

   Key Observation

   Index 14's entry became visible between crash time and dump time — the store buffer drained during signal handler execution, proving the write existed but was not yet observable to the crashing thread.

   Root Cause

   ARM64 store-visibility race in TLSIndexToMethodTableMap between lockless reader and lock-protected writer.

   Writer Path (under g_TLSCrst lock)

   // threadstatics.cpp, TLSIndexToMethodTableMap::Set() line ~240
   pMap[index] = rawValue;  // ← PLAIN STORE (no release barrier)

   Called from EnsureTlsIndexAllocated() → GetTLSIndexForThreadStatic() during type loading (MethodTableBuilder::SetupMethodTable2).

   Reader Path (NO lock)

   // threadstatics.h, TLSIndexToMethodTableMap::Lookup() line ~130
   TADDR pMap = VolatileLoad(&m_pMap);           // ← ldapr (acquire) for pointer
   TADDR value = VolatileLoadWithoutBarrier(&pMap[index]); // ← PLAIN LDR for entry

   Called from LookupMethodTableAndFlagForThreadStatic() → GetThreadLocalStaticBase() on the hot path.

   Why This Crashes on ARM64

   1. The lock doesn't help — only the writer acquires g_TLSCrst. The reader is lockless (hot path optimization).
   2. VolatileLoadWithoutBarrier = plain ldr on ARM64 — no acquire semantics, can return stale cache-line data.
   3. ARM64 weak memory model — a plain store on Core A is not guaranteed to be visible to a plain load on Core B without explicit barriers.
   4. The IsTlsIndexAllocated flag uses InterlockedOr (store) / VolatileLoad (acquire read), but the subsequent map lookup uses VolatileLoadWithoutBarrier which bypasses the ordering that the acquire established.

   Race Sequence

   Thread A (type loader):                    Thread B (class init):
   ─────────────────────────────────────────────────────────────────
   Lock(g_TLSCrst)
   pMap[14] = rawValue  [plain store]
   Unlock(g_TLSCrst)
   InterlockedOr(flags, TlsAllocated)
                                              VolatileLoad(flags) → sees TlsAllocated ✓
                                              VolatileLoad(&m_pMap) → gets pMap ptr ✓
                                              VolatileLoadWithoutBarrier(&pMap[14])
                                                → reads stale 0 ✗ (store not yet visible!)
                                              pMT = UnwrapValue(0) = NULL
                                              pMT->m_pEEClass  → SIGSEGV
   

   Origin

   • Introduced: Commit eb8f54d92b (2024-06-13) — PR Make normal statics simpler #99183 "Make normal statics simpler" by David Wrighton
     • Created TLSIndexToMethodTableMap with VolatileLoadWithoutBarrier in Lookup()
   • Made reachable: Commit 26612c84cad (2024-10-30) — PR Move static helpers to managed #108167 "Move static helpers to managed"
     • Added GetThreadStaticsByMethodTable QCall enabling parallel thread-local access during class init

   Disassembly Confirmation

   Set() — all stores are plain str (no stlr/store-release)

   ;; GROW PATH:
   +128: str x23, [x20]             ; pMap = newMap        — PLAIN STORE
   +132: str w26, [x20, #0x8]       ; m_maxIndex = newSize — PLAIN STORE
   
   ;; ENTRY WRITE:
   +168: ldr x9, [x20]              ; x9 = pMap
   +172: and x10, x19, #0xffffff    ; x10 = index
   +176: str x8, [x9, x10, lsl #3]  ; pMap[index] = rawValue — PLAIN STORE

   Lookup() (inlined in GetThreadLocalStaticBase) — entry read is plain ldr

   +68:  ldapr w10, [x10]           ; VolatileLoad(&m_maxIndex) — ACQUIRE ✓
   +80:  ldapr x8, [x8]             ; VolatileLoad(&pMap)       — ACQUIRE ✓
   +88:  ldr x8, [x8, x9, lsl #3]  ; pMap[index]               — PLAIN LDR ✗

   The reader acquires the pointer with ldapr, but reads the entry data with plain ldr. Without a matching stlr on the writer side, there is no happens-before relationship for the entry data on ARM64.

   Proposed Fix

   Option 1 (Minimal — fix writer): Change plain store to store-release in Set():

   // In TLSIndexToMethodTableMap::Set():
   VolatileStore(&pMap[index], rawValue);  // store-release ensures visibility

   Option 2 (Belt-and-suspenders — fix reader): Change to acquire-load in Lookup():

   // In TLSIndexToMethodTableMap::Lookup():
   TADDR value = VolatileLoad(&pMap[index]);  // load-acquire pairs with store-release

   Option 3 (Defense-in-depth): Add NULL check after LookupMethodTableAndFlagForThreadStatic():

   // In GetThreadLocalStaticBase(), after line 506:
   if (pMT == nullptr)
   {
       // Race condition: retry or fall back to locked path
       goto RETRY_LOOKUP;
   }

   Recommended: Option 1 + Option 3. The store-release fixes the root cause with minimal perf impact (store-release is cheap on ARM64). The NULL check provides a safety net against any remaining edge cases.

   Note on the grow path: Set() also has pMap = newMap and m_maxIndex = newSize as plain stores after memcpy. These should also use VolatileStore for correctness, even though this particular crash was NOT triggered by a grow (m_maxIndex=16 was the initial/only allocation).

   👍React with 👍2Reacted by Michael Simons and Jan Kotas
10. jkotas

    added

    tenet-reliabilityReliability/stability related issue (stress, load problems, etc.)Reliability/stability related issue (stress, load problems, etc.)

    on May 5, 2026
11. jkotas

    assigned

    davidwrighton

    on May 5, 2026
12. davidwrighton

    closed this as completedin #127843on May 13, 2026
13. davidwrighton

    added a commit that references this issue on May 13, 2026

    Fix a series of possible race conditions in thread static variable in…

    ...

    This issue will close once commit c3be9c8 is merged into the 'main' branch.

    c3be9c8
14. dotnet-policy-service

    removed

    untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

    on May 13, 2026
15. github-actions

    locked and limited conversation to collaborators on Jun 13, 2026
16. eiriktsarpalis

    added a commit that references this issue on Jul 15, 2026

    Fix a series of possible race conditions in thread static variable in…

    ...

    This issue will close once commit 456847d is merged into the 'main' branch.

    456847d