Repository navigation
docs: FutuRe SIGSEGV sighting #9 — a fourth gc_heap frame, and a CI gate that named the wrong cause - #3241
Conversation
… a CI gate that named the wrong cause MeshWeaver.Plugins run 33778900486 produced the ninth dump of the zeroed- MethodTable family, on the same runtime binary as sighting #8 (10.0.11, libcoreclr build-id 989b56df…) and at a FOURTH distinct frame: si_signo=11 si_code=1 (SEGV_MAPERR) si_addr=0x0 TRAPNO=14 ERR=0x4 CR2=0x0 RIP = libcoreclr + 0x5d89e7 = WKS::gc_heap::background_mark_simple1(…) + 0x827 49 8b 08 mov rcx,[r8] / 48 83 e1 f8 and rcx,~7 / 44 8b 09 mov r9d,[rcx] ← RCX = 0 Sightings #4-#8 read `mov ecx,[rax]` with RAX=0; this is the same three- instruction sequence through r8/rcx. That confirms the 2026-08-12 note's rule in the direction it was stated: the instruction plus register state is the portable fingerprint, the frame and the register allocation are not. The family now spans mark, plan and sweep plus a heap walk. Disposal was clean and the trace record is provably complete (both FAULT-BUDGET suppression lines belong to a different pid): every DISPOSE_DONE reads `teardown clean`, leakedIoLeaves=0, zero QUIESCE_LEAK, zero DIRTY_TEARDOWN. Recorded with it: the reporting failure. xUnit v3 reports a child killed by a signal through the SAME `Catastrophic failure:` banner it uses for an escaped exception, and `dotnet test` flattens both to exit 1 — so Plugins' classify-test-run.py printed its static remedy asserting an escaped exception, a DI resolve in a single-argument Subscribe(onNext), a straggler capture that did not exist, and a teardown gate as the fix. All four are false here. A core PR was authored against that invented cause before the dump in the same job's artifact was opened. Read the dump before believing a gate's prose. Base rate measured over 1,197 Plugin Catalog CI runs (7,362 failure-annotation rows, plus all 92 cancelled main runs): 5 occurrences, all MeshWeaver.FutuRe.Test, one on main — 0.74% of non-cancelled runs, 2.5% since onset. Two of the five ran against core commits containing #3072, so #3026's fix reduced the rate without closing it. The /sigsegv skill is updated to nine sightings, the fourth frame, and the register-independent form of the fingerprint. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🟢 Approval recommended
Only small documentation clarity nits were identified, with straightforward wording fixes suggested.
Pull request overview
Updates MeshWeaver’s native-crash forensics documentation and the /sigsegv triage skill to record FutuRe SIGSEGV sighting #9, including a newly observed gc_heap::background_mark_simple1 frame and a clarified, register-independent fingerprint for the “zeroed MethodTable header” crash family. This also documents a CI-gate misclassification failure mode so future investigations don’t chase an invented “escaped Rx exception” narrative when the host was actually killed by a signal.
Changes:
- Add a new “sighting #9” entry to the native-crash debugging guide, including the 4th distinct
gc_heapframe and updated base-rate measurements. - Update the
/sigsegvskill’s fingerprint guidance to be register-allocation-independent and bump the sighting count to 9.
File summaries
| File | Description |
|---|---|
| src/MeshWeaver.Documentation/Data/Architecture/DebuggingNativeCrashes.md | Adds the sighting #9 record and documents the CI-gate misclassification scenario and updated stats. |
| .claude/skills/sigsegv/SKILL.md | Updates the “zeroed MethodTable header” fingerprint to match the new frame/register pattern and refreshes the sighting summary. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| `Portal hosts (shard 1)`, runtime `10.0.11`, libcoreclr build-id `989b56df…` — **the same binary as | ||
| sighting #8**). The register allocation differs and nothing else does: |
| exception that escaped onto a non-test thread, and `dotnet test` flattens both to exit 1 — so the | ||
| child's real code (`139`) survives only in the banner's text. MeshWeaver.Plugins' |
Test Results (shard 3)346 tests 344 ✅ 41s ⏱️ Results for commit 7896507. |
Test Results (shard 1)459 tests 459 ✅ 1m 2s ⏱️ Results for commit 7896507. |
Test Results (shard 4)2 201 tests 2 201 ✅ 2m 38s ⏱️ Results for commit 7896507. |
Test Results (shard 2)578 tests 386 ✅ 2m 37s ⏱️ Results for commit 7896507. |
Test Results (shard 5) 4 files 4 suites 1m 37s ⏱️ Results for commit 7896507. |
Test Results (shard 0)243 tests 243 ✅ 2m 8s ⏱️ Results for commit 7896507. |
Test Results 15 files 15 suites 10m 46s ⏱️ Results for commit 7896507. |
|
⛔ Merge-queue steward: left dequeued — a job other than a test shard failed — a build or gate failure is never a flake.
Not a catalogued flake. Fix the failure, or — with run URLs, an issue and an assertion-message pattern — add it to |
meshweaver-cloud
Bot
added
the
queue-rejected
The ninth dump of the zeroed-MethodTable family, read off MeshWeaver.Plugins run
33778900486— and a fourth distinctgc_heapframe.The dump
Same runtime binary as sighting #8 (
10.0.11, libcoreclr build-id989b56df…, both taken from the dump's ownNT_FILE):Sightings #4–#8 read
8b 08=mov ecx,[rax]withRAX=0. This is the same three-instruction sequence throughr8/rcx. That confirms the 2026-08-12 rule in the direction it was stated — the instruction plus register state is the portable fingerprint; the frame and the register allocation are not — and it is why the page's own fingerprint row is widened here rather than kept as one literal opcode pair.Four disjoint functions are now on record against the same
10.0.11symbols, spanning mark, plan and sweep plus a heap walk:0x5cb171background_sweep()+0xa6110.0.10's0x5cb1e1)0x5cf24cplan_phase(int)+0x24fc0x5d5ab2find_first_object(…)+0x1320x5d89e7background_mark_simple1(…)+0x827Disposal was clean, and the trace record is provably complete — both
FAULT-BUDGETsuppression lines belong to a different pid, which is the precondition this page insists on before reasoning from an absence. EveryDISPOSE_DONEforpid=49849readsteardown clean — all pooled I/O joined, async dispose queue drained;leakedIoLeaves=0, zeroDISPOSE_QUIESCE_LEAK, zeroDISPOSE_DIRTY_TEARDOWN.🚨 Recorded with it: the CI gate named a cause it could not know
xUnit v3 reports a child killed by a signal through the same
Catastrophic failure:banner it uses for an exception escaping onto a non-test thread, anddotnet testflattens both to exit 1. Plugins'classify-test-run.pywas not reading the banner's text, so it printed its static remedy: an escaped exception … typically a DI resolve inside a single-argumentSubscribe(onNext)… the stack is in the teardown-straggler capture below … fix it by GATING the callback on the hub's teardown state.All four clauses are false for this run, and the shard's very next line was
(no teardown-straggler capture file found for this project).A core PR was then authored against that invented cause, quoting the gate's own remedy as its evidence, before anyone opened the dump sitting in the same job's artifact. That is the expensive half of this sighting and the reason it is written down. The gate is fixed in Systemorph/MeshWeaver.Plugins#1289.
Base rate, measured before anything was changed
Plugin Catalog CI, 1,197 runs over 2026-08-29 → 2026-09-03; every failed run's every failed job's annotations read (7,362 rows), plus all 92 cancelledmainruns to close the "crashed before cancellation" gap.5 occurrences, all
MeshWeaver.FutuRe.Test, one onmain(run33575831719) — 0.74 % of non-cancelled runs, 2.5 % since onset (2026-09-01T23:15). No other suite shows the signature. Two of the five ran against core commits containing #3072 (87 and 156 ahead of17ee6d9fa), so #3026's fix reduced the rate without closing it — consistent with this page's standing verdict that MeshWeaver supplies the workload, not the defect.Scope
Documentation and the
/sigsegvskill only — no product code. The skill goes to nine sightings, the fourth frame, and the register-independent form of the fingerprint, so the next reader compares the dereference rather than one opcode pair.Refs #613, #3026, #890.
🤖 Generated with Claude Code