Skip to content

The FutuRe exit=139 is one background-GC fault, three times — and the dump reads fine on macOS - #1004

Merged
rbuergi merged 4 commits into
mainfrom
docs/613-exit139-second-confirmation
Aug 9, 2026
Merged

rbuergi merged 4 commits into
mainfrom
docs/613-exit139-second-confirmation

Conversation

@rbuergi

@rbuergi rbuergi commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

MeshWeaver.FutuRe.Test exit=139 was root-caused on 2026-08-06 as a CoreCLR background-GC fault — WKS::gc_heap::background_sweep() reading a swept object whose MethodTable pointer is exactly NULL — and explicitly not the collectible-ALC use-after-unload shape that #613 spent weeks chasing. One sighting makes that a hypothesis. #613 was closed on 2026-08-09 05:48Z and reopened 70 minutes later precisely because nobody could tell whether the next crash was the same thing.

It is. Both exit=139s of 2026-08-09 — including the one the issue was reopened on — resolve to the same fault at instruction granularity. On runtime 10.0.10 the whole comparison collapses to one number, RVA 0x5cb1e1:

2026-08-06 (31083356138) 2026-08-09 06:59Z (31299547995) 2026-08-09 11:05Z (31309378803)
faulting RVA background_sweep() 0x5cb1e1 (+0xa61) 0x5cb1e1 (+0xa61)
instruction mov ecx,[rax], rax == 0 8b 08 = mov ecx,[rax], RAX=0x0 8b 08, RAX=0x0
TRAPNO/ERR/CR2 14 / 0x4 / 0x0 14 / 0x4 / 0x0 14 / 0x4 / 0x0
si_signo/si_code/si_addr 11 / 1 / — 11 / 1 (SEGV_MAPERR) / 0x0 11 / 1 / 0x0
mutator phase 44 ms into a fresh fixture 96 ms into a Mesh.Dispose() 476 ms into a fresh fixture

The mutator phase varies while the faulting instruction does not — and that is the finding. A background-GC sweep runs concurrently with whatever the mutator is doing, so an identical fault at three different phases is what this diagnosis predicts; a teardown-ordering defect is the opposite, reproducing at one phase by construction (which is how NodeAlcUnloadTeardownOrderingTest could pin the exit=134). Reading the phase of a single sighting is how this crash spent weeks framed as a teardown race — I made the same mistake mid-investigation off two of three dumps and the third corrected it, so the caveat is now written down next to the table.

Two things this settles, both of which had been re-litigated

The teardown work neither caused nor cured it. The crashing tree contains #967 (node ALCs unload at the end of teardown) and #996 (pending Rx timers no longer root disposed hubs). The crashing process's own _meshweaver-test-trace.log records all 22 DISPOSE_DONE lines as teardown clean — all pooled I/O joined, async dispose queue drained, zero DISPOSE_QUIESCE_LEAK, zero DISPOSE_DIRTY_TEARDOWN; the straggler log holds two first-chance records, both from a different project's process.

The crash is not where the title says. Two of the three land inside a fresh test — in the 11:05Z case the last trace record is INIT_MEM for a newly constructed fixture whose predecessor had finished tearing down cleanly 470 ms earlier, under roughly one gen2 collection per second. "Teardown SIGSEGV in …" describes one sighting of three.

I checked the trace's completeness before reading anything into those zeros, per the FAULT-BUDGET rule the doc already states: all four suppression windows in that file belong to a later project's pid, so the crashing process's record is whole. That is what makes "zero quiesce leaks" evidence rather than an absence.

The method is the other half of the value

The doc opened with "🚨 You cannot open a Linux dump on macOS", which is true only of the DAC-backed managed commands (clrstack, verifyheap) — and it sent the last investigation into a slow emulated container for a fact the core carries in plain sight. The 185 MB dump download was abandoned at ~30 KB/s during the exit=134 work; that is the cost this framing imposes.

The whole analysis above ran on a Mac in about ten minutes, in pure Python over the ELF core plus one build-id lookup against Microsoft's public symbol server: NT_SIGINFO for the kernel's verdict, NT_FILE for the load base, a scan of the PT_LOAD segments for the faulting ucontext (TRAPNO == 14, RIP inside libcoreclr) since NT_PRSTATUS records createdump's own signal handler, then RVA → .symtab of the separately-downloaded coreclr.debug. The shipped libcoreclr.so is stripped to nine exported symbols, so resolving against it fails and looks like the technique doesn't work — the debug file has ~31 800 and the hit is exact. That recipe is now the first section, with the container kept for what genuinely needs it.

Scope

Documentation only — no code change, and deliberately so. The fault is runtime-internal bookkeeping; the doc's existing "do not 'fix' this by turning off concurrent GC" guidance stands. What MeshWeaver contributes is the workload that provokes it (per-[Fact] mesh build + teardown with collectible NodeType assemblies), which is not something to suppress.

Testing

  • MeshWeaver.Documentation builds clean under -c Release -warnaserror.
  • DocumentationLinkIntegrityTest — 1/1 passed.

🤖 Generated with Claude Code

…p again

The 2026-08-06 dump analysis concluded that MeshWeaver.FutuRe.Test's exit=139
is a CoreCLR background-GC fault — a swept object whose MethodTable pointer is
exactly NULL — and explicitly not the collectible-ALC use-after-unload shape
that #613 had been chasing. One sighting is a hypothesis; #613 was closed and
reopened on exactly that ambiguity.

Run 31309378803 on main @ 8a5c8bf reproduced it, and it resolves to the same
frame, the same instruction and the same registers: WKS::gc_heap::background_sweep,
mov ecx,[rax] with RAX=0, si_addr=0x0, TRAPNO=14/ERR=0x4/CR2=0x0. That tree
contains both #967 and #996, so the teardown work neither caused nor cured it —
and the trace log shows all 22 disposals clean, no quiesce leak, no dirty
teardown, with the crash landing ~476ms into a FRESH fixture. It is not a
teardown crash; the issue title has been misleading the search.

Also records the route that produced it: the whole analysis runs on macOS in
pure Python over the ELF core plus one build-id lookup against Microsoft's
public symbol server. The doc previously opened with 'you cannot open a Linux
dump on macOS', which is true only of the DAC-backed managed commands and sent
the last investigation to a slow emulated container for a fact the core carries
in plain sight.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the native-crash debugging documentation to make macOS-first triage faster and to record evidence that the MeshWeaver.FutuRe.Test exit=139 crash is the same CoreCLR background-GC fault on two separate dates/runs.

Changes:

  • Adds a macOS-first workflow to identify the faulting frame and dereferenced address without using Docker/DAC tooling.
  • Reframes the container-based approach as needed only for managed/DAC commands.
  • Documents a second independent reproduction (2026-08-09) to confirm the fault signature matches the earlier incident.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md Outdated
Comment thread src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md Outdated
Adding the second 2026-08-09 dump — the one #613 was reopened on — makes it
three sightings of the same fault, and it corrects something I got wrong an
hour ago. I had written that the crash 'is not a teardown crash at all'
because two of two landed inside a fresh fixture. dotnet-3050 lands 96ms into
a Mesh.Dispose().

The faulting instruction is byte-identical in all three: RVA 0x5cb1e1,
background_sweep+0xa61, mov ecx,[rax] with RAX=0, CR2=0, ERR=0x4. So the
mutator phase varies while the fault does not — which is what a concurrent
background sweep predicts, and the opposite of a teardown-ordering defect,
which reproduces at one phase by construction. Reading the phase of a single
sighting is how this issue got framed as a teardown race for weeks; the RVA is
the thing to compare, and on 10.0.10 it is one number.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rbuergi rbuergi changed the title The FutuRe exit=139 is the same background-GC fault, twice — and the dump reads fine on macOS The FutuRe exit=139 is one background-GC fault, three times — and the dump reads fine on macOS Aug 9, 2026
…iew)

The section promises 'pure Python, no elfutils' and then names <BUILD_ID> as
an input without saying where it comes from — which is exactly the step that
sends a reader back to eu-readelf and defeats the point. It is the same
section-header walk the rest of the recipe already does, so it is a few lines,
not a dependency.

Also indents the fence under item 4 so the numbered list does not break, and
says out loud that the shipped libcoreclr.so has nine STT_FUNC symbols: that
lookup returning nothing reads as 'the technique does not work' rather than
'this file is stripped, fetch the .debug'.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 0)

685 tests   685 ✅  4m 56s ⏱️
 10 suites    0 💤
 10 files      0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 3)

   11 files     11 suites   4m 58s ⏱️
1 981 tests 1 878 ✅ 103 💤 0 ❌
2 112 runs  2 009 ✅ 103 💤 0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 5)

1 724 tests   1 532 ✅  5m 50s ⏱️
   11 suites    192 💤
   11 files        0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 2)

1 898 tests   1 896 ✅  6m 20s ⏱️
   11 suites      2 💤
   11 files        0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 4)

1 208 tests   1 198 ✅  8m 0s ⏱️
   11 suites     10 💤
   11 files        0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results (shard 1)

1 188 tests   1 188 ✅  7m 15s ⏱️
   11 suites      0 💤
   11 files        0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@github-actions

github-actions Bot commented Aug 9, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

   65 files     65 suites   37m 22s ⏱️
8 684 tests 8 377 ✅ 307 💤 0 ❌
8 815 runs  8 508 ✅ 307 💤 0 ❌

Results for commit 608b0cb.

♻️ This comment has been updated with latest results.

@rbuergi
rbuergi merged commit 9a53bcf into main Aug 9, 2026
16 checks passed
@rbuergi
rbuergi deleted the docs/613-exit139-second-confirmation branch August 19, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants