CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here)

CI run 30012798237 (PR #609, shard 3): MeshWeaver.FutuRe.Test segfaulted at/near process exit — every completed test PASSED, then Segmentation fault, exit=139. createdump wrote a dump successfully in-run ([createdump] Dump successfully written in 1685ms) — the run's artifacts/dump are the root-cause lead.

This is NOT the known ClrMD DAC TLS-destructor class (#497 / ClrMdDacPin): FutuRe.Test contains no ClrMD/DataTarget usage (repo grep: DataTarget only in Monolith.Test). Different teardown-crash class — prime suspect is the collectible-ALC unload family recently reshaped on main (#596 kernel collectible ALC, #605 ALC eviction on recompile; cf. the earlier hub-construction segfault race where an ALC was unloaded without evicting a static reflection cache).

PR #609's diff (managed Messaging.Hub NACK + OAuth store) has no native/ALC surface; the crash is treated as pre-existing and intermittent. Filing for its own root-cause investigation: pull the dump from run 30012798237's artifacts and resolve the faulting frame.

🤖 Generated with Claude Code

Root cause found from the core dump's teardown-straggler log (downloaded run 30016043561 shard3, dotnet-3028.dmp + teardown-stragglers.log):

The crash is a teardown race between a TaskPool-scheduled layout-area render and collectible-ALC unload, NOT a random native fault:

Captured straggler (1, on .NET TP Worker thread 7, 14:45:11): System.ObjectDisposedException: ... LifetimeScope ... already been disposed, thrown from NodeMenuItemsExtensions.RenderMenus (NodeMenuItemsExtensions.cs:86, host.Hub.ServiceProvider.GetService<ILoggerFactory>()) ← LayoutDefinition.Render (LayoutDefinition.cs:203) subscribed on the Reactive TaskPoolScheduler.
So a layout-area render pipeline is still executing on a ThreadPool thread AFTER Mesh.Dispose() disposed the Autofac ServiceProvider.
FutuRe.Test compiles NodeTypes (BusinessUnit / LocalAnalysis / GroupAnalysis) into collectible ALCs. Menu/graph renders touch NodeType-compiled types. The ObjectDisposedException is the early symptom; the exit=139 SIGSEGV is the same straggler thread executing IL from a NodeType collectible ALC that a post-disposal GC then unloads → instruction fetch into unmapped memory. Same class as the hub-construction ALC-unload race and Fix NodeType-compilation memory leaks (ALC-per-recompile + LSP workspace) #605.
The known TeardownStragglerCapturer drain (Orleans connection pumps) and the fix(teardown): classify disposal as cancellation at Task sinks; drain the mesh before the legacy host's container dies #231 mesh drain do NOT cover TaskPool-scheduled layout renders.

Fix in progress: quiesce in-flight layout-area renders (cancel + join the TaskPool render work bound to the mesh) BEFORE Mesh.Dispose() disposes the ServiceProvider / unloads ALCs — a framework/host disposal-ordering fix (not a test-only or null-guard band-aid). Native faulting frame being confirmed on Linux (dotnet-dump can't read a Linux dump on macOS). Deterministic repro + verification to follow.

🤖 Generated with Claude Code

Merge pull request #621 from Systemorph/fix/layout-render-teardown-drain

Merge pull request #645 from Systemorph/fix/teardown-hub-creation-freeze

Reopening — this recurred after #645. Fresh exit=139 on run 30330569833 (shard 3), dump dotnet-2910.dmp analysed. Confirmed the right process: MeshWeaver.FutuRe.Test.dll loaded, ClrMD absent (so not the deliberate-crash test).

The framing in this issue and in `HostedHubsCollection.CloseCreation`'s doc comment is wrong

Both describe a teardown / post-dispose race. The dump shows the crash on a BUILD path — MeshBuilder.BuildHub is at the bottom of the stack. It is fixture init, not teardown, which is why #645's CloseCreation guard (flipped when disposal begins) was never going to catch it.

MeshBuilder.BuildHub                          ← outer hub being CONSTRUCTED
└ MessageHubConfiguration.Build
  └ MeshDataSource.SubscribeToOwnDeletion     ← a .WithInitialization hook, NOT a teardown path
    └ ownStream.Subscribe(…)
      └ WorkspaceStreams.CreateReducedStream
        └ Lazy → HostedHubsCollection.CreateHub   ← hub #2 built RE-ENTRANTLY
          └ MessageHub..ctor → WithTypeAndRelatedTypesFor
            └ XmlDocs.Summary(Type) → Namotion ToXmlDocsContent
              └ StringBuilder.ToString() → SIGSEGV

Note the method name is misleading: SubscribeToOwnDeletion names what the subscription later watches for, not when it runs.

Ruled out by the dump — please don't re-run these

Hypothesis	Killed by
Post-dispose / ALC-unload teardown race	Crash is on `BuildHub`; no thread unloading an ALC or emitting
Per-ALC duplicate statics splitting the `XmlDocs` lock	Exactly one copy of `MeshWeaver.Messaging.Hub` and one of `Namotion.Reflection` across 130 modules
Concurrent XLinq enumeration tearing the cached `XDocument` (what the `XmlDocs` lock exists to prevent)	Only one thread is in Namotion at all — no concurrency to serialise
GC heap corruption, XmlDocs the victim	`verifyheap` clean; heap 183 MB; no memory spike (last delta +8 MiB)

So: a single-threaded, non-concurrent, non-corrupt SIGSEGV inside an allocation on the only Cooperative/(GC) thread. Root cause still unknown — remaining candidates are Namotion/XLinq internals or a stack condition on that ~25-frame re-entrant chain.

Next step

Remove the re-entrancy. MessageHubConfiguration.Build runs SyncBuildupActions synchronously inside Build, before StartMessageProcessing() — so an init hook that subscribes can build a second hub mid-construction. Deferring that subscribe past StartMessageProcessing is defensible on its own merits and removes the chain the process dies on.

Instrumentation landed in #700 (the bare catch {} around this whole path now logs), so the next occurrence should carry a named cause. Analysis recipe: Doc/Architecture/DebuggingNativeCrashes.

Three fresh data points for the all-tests-pass-then-host-exits-non-zero signature, from CI runs on #708 (2026-07-29). Adding them because the crashing project set varies run to run, which constrains what the cause can be.

run	crashed with `exit=2`	test results in that project
`fb96e2c4c`	`AI.Test`, `PluginCatalog.Test`	919 pass / 54 pass, 0 failed
`16da70383`	`AI.Test`, `Acme.Test`	919 pass / 76 pass, 0 failed

In every case the project reports Total: N, Errors: 0, Failed: 0 and then the host exits non-zero — the shard fails on the exit-code gate, not on any test.

All three reproduce clean locally on the same commit, -c Release:

PluginCatalog.Test — 54/54, exit 0
Acme.Test — 76/76, exit 0
(Hosting.Monolith.Test — 421 tests, 0 failed, exit 0)

So it is CI-environment/load dependent, not a semantic failure, consistent with this issue's teardown-crash class rather than anything in the changes under test. Two further constraints from the pair of runs above: the set is not stable across otherwise-identical runs, and the second run's only new content was a Concat → Merge change in the compile source snapshot — present in the run where PluginCatalog stopped crashing and Acme started, so it does not explain either.

Worth noting these are exit=2, not the exit=139 in this issue's title — same "passed everything, died on the way out" shape, different exit code. If those are separate mechanisms this belongs in its own issue; flagging rather than assuming.

Caveat on attribution: main's own runs at c7245ca24 had AI.Test exit=0 and PluginCatalog.Test exit=0, so I cannot claim my branch is identical to main here — only that no test fails, all three pass locally, and the varying set argues against a deterministic cause in the diff.

Local repro of the post-dispose hub construction chain — the mechanism this issue's dump analysis named, now reproducible on a dev box (previously recorded as "no local repro exists").

Run MeshWeaver.AI.Test the way CI does — native host, CWD = bin dir:

cd test/MeshWeaver.AI.Test/bin/Release/net10.0
dotnet MeshWeaver.AI.Test.dll -trx TestResults/native.trx
grep -c FIRST-CHANCE test-logs/teardown-stragglers.log     # → 19

919 tests pass, exit 0, and TeardownStragglerCapturer records 19 first-chance ObjectDisposedExceptions, 14 of them this exact chain:

MeshQuery.MergeProviderObservables (MeshQuery.cs:531/589)   ← a query emission still live AFTER disposal
 → MonolithRoutingService.CreateHub (MonolithRoutingService.cs:175)
   → HostedHubsCollection.GetHub → CreateHub (HostedHubsCollection.cs:140)
     → MessageHubConfiguration.Build (MessageHubConfiguration.cs:413)   ← SyncBuildupActions, pre-StartMessageProcessing
       → MeshDataSourceExtensions.SubscribeToOwnDeletion (MeshDataSource.cs:959)
         → NodeTypeCompilationHelpers.InstallReleaseRequestWatcher (:761)
           → AutofacServiceProvider.GetService → disposed LifetimeScope → throw

What this pins. The trigger is not teardown ordering inside a hub — it is that a live MeshQuery emission survives mesh disposal and constructs a NEW hub. MonolithRoutingService.CreateHub runs off a reactive emission, HostedHubsCollection builds the hub, and Build's synchronous buildup resolves services from a container that is already gone. Locally these are caught (first-chance only), so exit stays 0; on CI the same construction is what walks the TypeRegistry while a collectible ALC is unloading.

That matches the "defer SubscribeToOwnDeletion past StartMessageProcessing" direction already noted here, but suggests it is not sufficient on its own: deferring the hook moves when the resolve happens, while the real defect is that hub creation is attempted at all after disposal has begun. A creation gate — HostedHubsCollection.GetHub/CreateHub refusing once disposal starts, and the requesting stream going dead rather than getting a half-built hub — addresses the trigger rather than one of its consumers.

Not the same shape as the other teardown stragglers, worth separating:

project	stragglers	chain
`AI.Test`	19	post-dispose hub construction (above)
`Hosting.Monolith.Test`	12	`MeshQuery` / `SynchronizationStream` / `MonolithRoutingService`
`PluginCatalog.Test`	2	`MeshService.Query` / `MeshNodeCompilationService`
`Acme.Test`	23 → 0	`XUnitFileLogger.GetMinLogLevel` resolving from the disposed container — fixture-only, fixed in #708

The Acme one was a distinct, unrelated defect (the test logger held the container and resolved per log call, so anything logging from a Dispose path threw); fixing it does not touch the three above.

Caveat: all of these are caught first-chance exceptions, so none of them is proven to be the exit=2 seen on CI. They are the same post-disposal-access class this issue tracks, and the AI chain is the one that matches the dump.

Hand the test logger a resolved config, not the container it outlives

The post-dispose hub-construction trigger is fixed in #715 — measured, with a deterministic repro.

Per the maintainer's teardown contract (properly finish the requests we started, transparently stop assuming new requests): CloseCreation (#635) already handles the refuse-new half; #715 closes the finish half — the "check-then-act residue" #488 identified. DisposeHubsReactive's completion join now waits for in-flight constructions (tracked with a counter + completion ping around the creation Lazy) and disposes whatever a late construction produced before signalling DisposalCompleted, so the container outlives every in-flight Build.

Evidence:

Deterministic repro (InflightHubCreationDrainTest): construction parked inside WithInitialization while the parent disposes — fails without the fix (disposal completed in ~20 ms mid-construction), passes with it.
MeshWeaver.AI.Test, native host: the hub-construction straggler chain (the stack in my previous comment) goes 14 → 0. 919/919 green before and after.

Still open, and now the dominant remaining shape (12 stragglers in the same run, all ObjectDisposedException, all caught):

MeshWeaver.AI.ThreadExecution.*            (12)
  AgentChatClient.GetStreamingResponseAsync (6)
  MeshQuery.*                               (6)
  Workspace.GetRemoteStream* / GetWorkspace (2 each)

That is in-flight agent-round work — not hub construction — reading queries / remote streams after disposal began: the consumer-side class, needing its own finish-or-refuse treatment at the round boundary (ThreadExecution), not another change in the hub layer.

Also for the record: the exit=2 CI shard failures remain unexplained — attempt 2 of the same commit ran clean with no code change, so they are transient and were never tied to any of these caught exceptions.

Correction to my earlier note that "the exit=2 CI shard failures remain unexplained" — solved, and it is a distinct defect from this issue's caught-straggler class: #717.

exit=2 is xUnit v3's AppDomain.UnhandledException handler calling Environment.Exit(2). The handler reports [FATAL ERROR] <exception> through the reporter, then finishEvent.Wait()s for the run to END — which is why the trx is all green and the summary prints Errors: 0 — then exits 2. (Diagnostic tip: grep the shard log for FATAL ERROR, not "catastrophic"; and TeardownStragglerCapturer's unhandled hook never fires because xUnit's handler subscribed first and Environment.Exit kills the process before later handlers run.)

The thrower, from #716's failing run:

[FATAL ERROR] System.ObjectDisposedException
  at AutofacServiceProvider.GetService
  at MeshService.CaptureContext      (MeshService.cs:62)
  at MeshService.Query[T]            (MeshService.cs:257)
  at MeshNodeCompilationService.DirectSourceProbe b__2 (:618)
  at CombineLatest.Run ← SubscribeRaw ← DelaySubscription ← ThreadPool

The probe's per-leg mesh.Query(...) evaluates inside CombineLatest's subscription, which DelaySubscription runs on a ThreadPool tick; mesh disposed in between → synchronous throw during a Producer's subscribe → routed to the subscribe caller (the scheduler), not OnError — so every .Catch in the chain is blind to it → unhandled. #717 wraps each leg in Observable.Defer, whose factory exceptions ARE forwarded to OnError, landing in the leg's existing .Catch.

Timeline fits perfectly: the transients began when #690 put the probe on every compile; #708 attempt 2 passing on the identical commit was winning the timer race, not absence of the bug.

Fresh occurrence, new exit code for the ledger: main run 31157208644 (commit 0a01185, the #892 merge), shard 2 — MeshWeaver.FutuRe.Test exit=134 (SIGABRT, 'timeout: the monitored command dumped core') after all tests passed. Same passed-everything-died-on-the-way-out signature; the class ledger now spans exit=139 (SIGSEGV), exit=2 (xUnit UnhandledException → #717, fixed), and exit=134. A rerun of the same commit went green, so it remains load/timing-dependent. Core dump was written in-run — run 31157208644's artifacts are the next analysis lead if this recurs.

🤖 Generated with Claude Code

Maintainer direction for the next occurrence (2026-08-07):

"if future after fix of dispose => analyze core dump. we saw we subscribe to finish disposing"

Two things to carry forward:

Do not treat a recurrence as a flake — pull the core dump and resolve the faulting frame. createdump has written a dump in-run on past occurrences (run 30012798237: "Dump successfully written in 1685ms"), and the dump is what produced every real answer in this issue's history; the guesses did not. Fresh occurrence logged earlier today on main run 31157208644 shard 2 — MeshWeaver.FutuRe.Test exit=134 (SIGABRT, "timeout: the monitored command dumped core") after every test passed. The class ledger now spans exit=139 (SIGSEGV), exit=2 (xUnit UnhandledException → Defer the probe's query legs: a subscribe-time throw was killing the test host #717, fixed) and exit=134.
The lead: we SUBSCRIBE to disposal completion. Teardown awaits hub.DisposalCompleted (e.g. MonolithMeshTestBase and the leak probes do DisposalCompleted.Catch(...).FirstOrDefaultAsync().Timeout(30s).ToTask()). That subscription's continuation runs at the moment disposal finishes — i.e. exactly when the collectible ALC is being unloaded and the Autofac LifetimeScope is going away. A continuation scheduled off that signal is therefore the prime candidate for post-dispose work executing on an unloading ALC, which is the documented mechanism for this crash family (instruction fetch into unmapped memory after the ALC unloads).

So the next investigation should start from: who subscribes to DisposalCompleted, on which scheduler does that continuation run, and what does it touch? — rather than re-scanning teardown ordering generally.

Related in-flight work that changes the surrounding timing and may affect reproduction: PR #910 moves in-process change-feed fan-out off the publisher's thread onto a dedicated serial dispatch loop.

🤖 Generated with Claude Code

Root cause: a teardown phase inversion — node ALCs unload before the pooled work over their types is joined

Established from CI run 31157208644 attempt 1, job 92801274946 (shard 2), which printed a fully symbolized managed Fatal error. stack before createdump ran. That stack — not the 185 MB dump — is the evidence.

System.AccessViolationException: Attempted to read or write protected memory.
  at System.Runtime.CompilerServices.StaticsHelpers.GetGCThreadStaticBase(MethodTable*)
  at System.Collections.Generic.SegmentedArrayBuilder`1.Expand(Int32)
  at System.Linq.Enumerable.<ToArray>g__EnumerableToArray|324_0
  at MeshWeaver.Data.Workspace+<>c__11`1.<GetStream>b__11_0(ChangeItem`1<EntityStore>)
  ... Rx layout-render chain (ReactiveExtensions.ThrottleImmediate ×2) ...
  at MeshWeaver.Mesh.Threading.IoPool+<SubscribeThroughPool>b__1>d.MoveNext()

The ordering:

MeshDataSource.cs:879 (in SubscribeToOwnDeletion, :840) unloads a node's collectible ALC from a synchronous hub-disposal callback: hub.RegisterForDisposal(_ => compilationCache.UnloadNodeContexts(sanitizedNodeName));. That runs in DisposeImpl() (MessageHub.cs:1835) — strictly before SignalDisposalCompleted() (MessageHub.cs:1855).
In-flight layout-render subscribes are only cancelled and joined after DisposalCompleted: MonolithMeshTestBase.cs:1317 (await) → :1326 (ioPools?.DrainAll()); same order in MeshTeardownExtensions.cs:114 → :126.
Between those two points a tracked Layout-pool leaf (LayoutAreaHost.ScheduleRenderSubscribe → IoPool.SubscribeThroughPool, IoPool.cs:188) is still inside source.Subscribe(observer), running Workspace.cs:209's .Cast<T>().ToArray() where T is a NodeType-compiled type. ToArray → SegmentedArrayBuilder<T> reads that collectible type's GC thread-static base out of the freed LoaderAllocator → AV → SIGABRT (exit=134).

This violates the framework's own written contract. MeshTeardownSignal's XML doc says the terminal signal is where "everything that must not run before teardown truly ends — disposing the service scope, unloading node ALCs, starting the next test's mesh" subscribes, and LayoutAreaHost.cs:843 promises DrainAll() joins the render subscribe "BEFORE the scope disposes / the ALC unloads". Two individually-correct changes — pool-tracked render subscribes, and per-node ALC reclaim for the CI OOM — compose into this bug.

Proof of the ordering that needs no crash repro: NodeTypeAssemblyLeakTest.NodeTypeAssemblyContext_IsCollected_AfterMeshDisposal (green on main) disposes, awaits DisposalCompleted, then asserts zero surviving DynamicNode_* ALCs — i.e. it pins the unload as complete before DrainAll() runs.

Ruled out (with reasons)

CloseCreation / post-dispose hub construction (the standing hypothesis, and Patch ack by write identity, not emission counting (second acked-write-loss mode) #635/Finish in-flight hub constructions at disposal instead of racing them #715/Fix re-entrant hub construction (FutuRe.Test SIGSEGV / exit=139) #774): the faulting stack contains no BuildHub/CreateHub frame.
A leaked or untracked pool leaf: the crashing thread is inside SubscribeThroughPool's gated body, so it holds a permit — DrainAll() provably had not run. It is ordering, not a tracking gap.
Live recompile eviction: the only non-test node-ALC unload triggers are MeshDataSource.cs:879 and CompilationCacheService.Dispose() (the latter correctly ordered, at scope disposal).
The 2026-08-06 exit=139 verdict in DebuggingNativeCrashes.md ("MT pointer exactly 0 ⇒ CoreCLR background-GC bug, not use-after-unload") does not apply here: this is exit=134, an AV on a non-null but unmapped LoaderAllocator — exactly the use-after-unload shape that doc distinguishes. That note should be scoped to the exit=139 sighting.

Not reproducible locally

3/3 local dotnet test test/MeshWeaver.FutuRe.Test -c Release runs exited 0 on main @ c8ca67a09. This class is CI/load-dependent, consistent with the issue history.

The fix, and why it needs care

"Defer the unload past DrainAll" is right at mesh teardown but cannot be unconditional: in prod the mesh never tears down, so a node hub disposing on its own would leak its ALC for the process lifetime — reintroducing the CI-OOM leak that :879 was added to fix. So it needs a "mesh is tearing down" predicate at the node hub's dispose callback (IMessageHub.IsDisposing on the mesh hub, captured at construction, is the existing idiom — MonolithRoutingService.cs:73), and NodeTypeAssemblyLeakTest must move its wait from DisposalCompleted to the terminal teardown signal, because that test currently pins the buggy ordering as the contract.

Cheapest confirmation, no SIGSEGV needed: a deterministic ordering test — record the invocation order of ICompilationCacheService.UnloadNodeContexts vs IoPoolRegistry.DrainAll() during a monolith mesh teardown (decorate the cache service in ConfigureServices). It fails on main today.

CD failed on main: run 33057922048 for commit 2e2e41ca70e25583b544f2d82bfc58c4ea74ba8d (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

🔁 CD reconcile attempt 1/3 for c567c3d (run).

main's HEAD is missing part of its image set in ACR (that run's annotations name which repositories), so this run is publishing the full set for c567c3d. The commit whose CD run originally failed is NOT re-published — it is superseded by HEAD on purpose: the version tag comes from the BUILDING run's number, so re-publishing older code would mint a higher -ci.<n> for it and roll every install backwards.

✅ Healed c567c3d (run) — main's HEAD now has the complete image set (portal 3.0.0-rc8.ci.6081), verified against ACR. Installs pick it up on their next 6 h self-update poll. Close this issue if nothing else is outstanding.

CD failed on main: run 33059618095 for commit c567c3db3b7b0c4e06f3aac2343acdcc45264f19 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33063843072 for commit 0596387d4e1f2095c3aa1d4d8df61af1f4331556 (entered via reconcile).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

CD failed on main: run 33062743687 for commit 0596387d4e1f2095c3aa1d4d8df61af1f4331556 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

Run 33063843072 (the scheduled reconcile on 0596387d) — triaged: the bake published (sealed on all three targets, identity sbe62e87…); the job then died in "Assert the armed release is one an environment can be rolled to" because RELEASE_VERSION reads needs.portal-image.outputs.version, which is empty on a bake-only run (that job is skipped). It also wrote no release marker. This was the first reconcile that ever fired (the cron was starved until #2491), so the assumption had never been exercised. Fix: #2496 — publish-bake resolves the version once (this run's, else the promoted image's memex-portal-ai:<version> tag by sha), pinned by a PlatformBakeLaneGuard test.

CD failed on main: run 33064800272 for commit d02ca8847c680ddde7cc493ef9da74cb896126e5 (entered via push).

Triage the failed job in the linked run - which of the two things CD delivers is broken depends on WHICH job went red:

an IMAGE job / promote / verify-images: no complete image set was published. Publication is all-or-nothing, so every self-updating install stays on the PREVIOUS image until a CD run promotes the full set.
preflight / notify-platform-update: the images may be fine, but the release EVENT did not reach the control instance, so no node repo re-bakes against this commit until its own schedule poll (Release broadcast has never delivered: notify-platform-update succeeds, Hosting/PlatformBuilds stays empty, no satellite dispatch since 08-22 #2235). Read the job's ::error:: line - it names what to provision or what refused the delivery.

The hourly reconciler will re-attempt main's HEAD on its own (bounded at 3 attempts per commit, logged here); you do NOT need to force a merge to trigger delivery. Close this issue once CD is green.

fix: quote skill front matter, and guard it — three skills on main do…

Closing as not planned, for the reason the title already records: this is an upstream CoreCLR defect (an object's MethodTable word zeroed; the GC faults walking it) — not fixable in this repository. The teardown-hardening work that reduces exposure to the class has its own issues and largely landed (#2527/#2538/#2549/#2598); a recurrence with a fresh dump would be a new investigation, not this thread.

Housekeeping note: this issue also accumulated the CD ci-failure ledger by label (184 bot comments). Closing it does not break that ledger — main-cd.yml selects any open ci-failure-labeled issue and creates a fresh one when none is open; the ledger is currently landing on #2500 (which carries the label). The per-commit attempt counts recorded here are historical only.

fix(query): teardown cancellation terminates a query walk instead of …

Merge pull request #2736 from Systemorph/fix/613-filesystem-iopool-re…

Two more dumps read (2026-09-07 and 2026-09-08) — and as a pair they close the `si_addr` and the "the frame is migrating" questions

Recurrence triage on a fresh Portal hosts (shard 1) red
(Plugins run 34210183539,
job 102009955282). Per the disposition on Plugins#1347 I am adding sightings here rather than
opening a rival Plugins bug. Both dumps were read; neither cause was named before reading one.

The pair

	#11 `MeshWeaver.GitS-37600.dmp`	#12 `MeshWeaver.Futu-50607.dmp`
run / job	`34069990582` / `101585455000`	`34210183539` / `102009955282`
when / branch	2026-09-07 00:29Z, `main`	2026-09-08 09:28Z, `fix/1390-teardown-resolve-guard`
suite	`MeshWeaver.GitSync.Test`	`MeshWeaver.FutuRe.Test`
`si_signo` / `si_code`	11 / 1 (`SEGV_MAPERR`)	11 / 1 (`SEGV_MAPERR`)
`si_addr` / `CR2`	`0x4`	`0x0`
`TRAPNO` / `ERR`	14 / `0x4`	14 / `0x4`
faulting RVA	`0x373887`	`0x5cb171`
frame	`LCGMethodResolver::GetCodeInfo(…)+0x1f7`	`WKS::gc_heap::background_sweep()+0xa61`
thread	mutator, inside the JIT	dedicated background-GC thread, no managed frame
MT field read	`m_BaseSize` (offset 4)	`m_dwFlags` (offset 0)
the word at `[R15]`	reads zero	reads zero
runtime / build-id	`10.0.11` / `989b56df…`	`10.0.11` / `989b56df…`

#11  0x373884: 49 8b 07     mov  (%r15),%rax      ; MethodTable word -> 0
     0x373887: 8b 70 04     mov  0x4(%rax),%esi   ; <-- FAULT at 0x4   (MT->m_BaseSize)

#12  0x5cb16a: 49 8b 07     mov  (%r15),%rax      ; MethodTable word -> 0
     0x5cb16d: 48 83 e0 f8  and  $-8,%rax         ; strip GC mark bits
     0x5cb171: 8b 08        mov  (%rax),%ecx      ; <-- FAULT at 0x0   (MT->m_dwFlags)

In both the null is confirmed at its source — the bytes at R15 read zero — not inferred from
RAX.

What the pair settles that one dump could not

si_addr is not part of the fingerprint. 0x4 and 0x0 are just the offsets of m_BaseSize
and m_dwFlags, measured 33 hours apart against the same runtime binary. Sighting [Snyk] Upgrade @microsoft/signalr from 3.1.9 to 3.1.31 #10 argued
this; this pair measures it.
The frame revisits — it is not a progression. 0x5cb171 is not new: sighting [Snyk] Upgrade react-dropzone from 12.0.5 to 12.1.0 #9's own table
lists it as background_sweep()+0xa61, first seen 2026-08-18. So reading [Snyk] Upgrade @microsoft/signalr from 3.1.9 to 3.1.31 #10's departure from
gc_heap as the family migrating toward the LCG/serialization workload was over-reading a sample
of one. I have marked that consequence bullet in the doc as superseded in part.
Culprit vs victim, from two directions at once. Mesh weaver #12 faults on a dedicated BGC thread with no
managed frame (ThreadEntry → CreateSuspendableThread → bgc_thread_stub → bgc_thread_function → gc1 → background_sweep); Parsing Storage for Cashflow Import #11 faults on a mutator inside the JIT (JitCompileCode → JitCompileCodeLockedEventWrapper → JitCompileCodeLocked → UnsafeJitFunction → CEECodeGenInfo → getMethodInfoWorker → GetCodeInfo). A defect in either path cannot produce the other; a heap
whose object headers can read as zero produces both.

Disposal clean, records provably complete, ALC hypothesis untouched

FAULT-BUDGET suppression lines belong to a different pid in both artifacts (2828 vs 37600;
2791 vs 50607), so neither crashing process's trace record has gaps. Every DISPOSE_DONE reads
teardown clean — all pooled I/O joined, async dispose queue drained (61/61 and 30/30),
leakedIoLeaves=0, zero DISPOSE_QUIESCE_LEAK, zero DISPOSE_DIRTY_TEARDOWN. alc=1 in
both — only the default AssemblyLoadContext was live, so there was no collectible context to unload
out from under anything. Neither job log contains Unwind: exception type, so the managed-exception
route through createdump is ruled out rather than assumed away.

Base rate, re-measured

Plugin Catalog CI, 600 runs over 2026-09-06T06:08Z … 2026-09-08T09:52Z; 10 excluded (job
listing unreadable). Denominator 343 = runs where at least one Portal hosts (shard N) job
reached a terminal conclusion (the rest were cancelled first — 311 of 600 are cancelled). All 59
failed Portal hosts jobs had their annotations read; 0 read failures.

4 occurrences — 1.17 % — all Portal hosts (shard 1):

when	run	branch	suite
2026-09-06 17:14Z	`34047985756`	`main`	`FutuRe.Test`
2026-09-06 20:15Z	`34057413159`	`fix/edu-union-wait-measures-behaviour`	`FutuRe.Test`
2026-09-07 00:29Z	`34069990582`	`main`	`GitSync.Test` (#11)
2026-09-08 09:28Z	`34210183539`	`fix/1390-teardown-resolve-guard`	`FutuRe.Test` (#12)

🚨 Not comparable to the 0.74 % above and not a trend — that used non-cancelled runs as its
denominator, this uses runs whose shard reached a verdict. What they agree on is the shape: a
low-single-digit-percent, shard-1-only rate that has not gone away, main included in both
windows. Three of the four were previously unrecorded anywhere.

Controls, because #10's first read was an impossible frame

RVA 0x5cb171 ↔ file offset 0x5ca171 (PT_LOAD maps vaddr 0x1c9680+ at 0x1c8680+) — the
0x1000 trap; symbols indexed by vaddr, bytes read by file offset.
RIP is an instruction boundary (the sequence starts at 0x5cb16a); 0x372887 in [Snyk] Upgrade @microsoft/signalr from 3.1.9 to 3.1.31 #10's first
read was not.
Core bytes at RIP == stock 10.0.11 libcoreclr.so bytes at the corresponding file offset.
Build-id read out of libcoreclr as mapped in each crashed process = 989b56df… = the build-id
of the .debug used to resolve.
Exactly one ucontext per core matches TRAPNO=14 ∧ RIP ∈ libcoreclr ∧ CR2 == si_addr.

Status — unchanged, deliberately

This does not reopen anything. It is the same zeroed-MethodTable family this issue owns and records
as upstream; nothing in either dump names a MeshWeaver defect, and no fix is proposed. Also measured,
so the next reader does not have to: MeshWeaver.FutuRe.Test has no ProjectReference to
MeshWeaver.AI (directly or via MeshWeaver.Hosting.Monolith.TestBase), so AI-engine teardown work
cannot reach it; and the only failure annotations on the shard-1 job in 34210183539 are the SIGSEGV
verdict and the generic exit code 1, so this is independent of the platform-pin staleness reddening
Plugins PRs.

What would actually move it from "upstream" to a named root cause — stated because I have not
done it and am not claiming it: run the affected suites under a runtime with heap verification
(DOTNET_HeapVerify) or a checked/debug CoreCLR, which catches the writer that zeroes the header
rather than the reader that trips over it. Every dump so far, this pair included, observes the read
side only, and no number of read-side dumps can distinguish "the runtime zeroed it" from "something
in the process wrote over it".

Conserved in Doc/Architecture/DebuggingNativeCrashes (sightings #11/#12) — the dumps expire
2026-09-14/15 and nothing above depends on them surviving.

docs(sigsegv): sightings #11/#12 — two dumps, two frames, one zeroed-…

Navigation Menu

Repository navigation

CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613

The framing in this issue and in `HostedHubsCollection.CloseCreation`'s doc comment is wrong

Ruled out by the dump — please don't re-run these

Next step

Root cause: a teardown phase inversion — node ALCs unload before the pooled work over their types is joined

Ruled out (with reasons)

Not reproducible locally

The fix, and why it needs care

216 remaining items

Two more dumps read (2026-09-07 and 2026-09-08) — and as a pair they close the `si_addr` and the "the frame is migrating" questions

The pair

What the pair settles that one dump could not

Disposal clean, records provably complete, ALC hypothesis untouched

Base rate, re-measured

Controls, because #10's first read was an impossible frame

Status — unchanged, deliberately

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613

Description

Activity

rbuergi commented on Jul 23, 2026

rbuergi commented on Jul 28, 2026

The framing in this issue and in HostedHubsCollection.CloseCreation's doc comment is wrong

Ruled out by the dump — please don't re-run these

Next step

rbuergi commented on Jul 29, 2026

rbuergi commented on Jul 29, 2026

rbuergi commented on Jul 29, 2026

rbuergi commented on Jul 29, 2026

rbuergi commented on Aug 7, 2026

rbuergi commented on Aug 7, 2026

rbuergi commented on Aug 8, 2026

Root cause: a teardown phase inversion — node ALCs unload before the pooled work over their types is joined

Ruled out (with reasons)

Not reproducible locally

The fix, and why it needs care

216 remaining items

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

rbuergi commented on Aug 27, 2026

github-actions commented on Aug 27, 2026

rbuergi commented on Aug 28, 2026

rbuergi commented on Sep 8, 2026

Two more dumps read (2026-09-07 and 2026-09-08) — and as a pair they close the si_addr and the "the frame is migrating" questions

The pair

What the pair settles that one dump could not

Disposal clean, records provably complete, ALC hypothesis untouched

Base rate, re-measured

Controls, because #10's first read was an impossible frame

Status — unchanged, deliberately

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

The framing in this issue and in `HostedHubsCollection.CloseCreation`'s doc comment is wrong

Two more dumps read (2026-09-07 and 2026-09-08) — and as a pair they close the `si_addr` and the "the frame is migrating" questions