Repository navigation
#2778 has recurred: a minted OwnerDisposing NACK reaches a waiter whose watch is still armed after the ShutDown registrant ran — 2 sightings, the catch-path hypothesis eliminated #4023
Description
Summary
NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCaller
has now failed twice, and its own assertion names the cause: #2778's defect is back. That
issue is CLOSED and nothing open tracks the recurrence.
The two sightings
| # | when | where |
|---|---|---|
| 1 | recorded in #3965's doc | run 34513634943, shard 4 |
| 2 | 2026-09-11T09:14:14Z | run 34581527040, job 103206660798, Portal hosts (shard 2), on MeshWeaver.Plugins PR #1642 |
🚨 Not the diff. #1642 changes a workflow and a Python comparator script — it cannot reach hub
teardown. Same argument as the pod-hub race: a diff that provably cannot touch the code failed the
test.
The assertion, in full
Did not expect collection {"JX3lvnrHzkybl5Z-NJ4rNQ"} to contain "JX3lvnrHzkybl5Z-NJ4rNQ" because
the owner minted anOwnerDisposingNACK for this patch, and a caller was armed and waiting for
it; the verdict must reach that watch even though the parent is pastDisposeHostedHubs. The
owner is now terminally Dead, so its ShutDown-phase registrant has already run — a watch still
armed here is the #2778 defect, and the caller then hears nothing and burns the whole 31 s
verdict budget for an answer that had already been minted.
🚨 The first hypothesis is ELIMINATED — measured, not argued
Doc/Architecture/DisposalStallVerdicts (written by #3965) names the hypothesis to eliminate first:
HandleShutdownCore reaches Dead on three paths — success, catch, and a finally backstop —
and only the first guarantees the ShutDown-phase registrant ran. It names the discriminator: the
catch's Error during shutdown of hub {address} line.
That line appears 0 times in the failing job's log (126,780 bytes, ANSI-stripped before
grepping). So the catch path did not run, the owner reached Dead cleanly, and the
ShutDown-phase registrant DID run.
Which leaves the remaining explanation, exactly as the assertion states it: the registrant ran and
the watch is still armed afterwards. That is #2778's shape, not a teardown-path variant.
This also sharpens the family distinction #3965 drew, and it is worth restating because conflating
these two has already cost time:
- A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510 — no verdict is ever minted. Fixed (fix(#3510): an install OWNS its package root — a recycle aimed at it waits #4009, merged).
- this — a verdict IS minted and the waiter never observes it. Open, and now twice-seen.
Why it matters beyond a red shard
The caller burns its entire 31 s verdict budget waiting for an answer that had already been
minted. In CI that is one red test. In production it is a write that was answered and a caller that
never heard — the silent-loss shape, where nothing is red and nothing is logged.
What a fix needs
- Find where the watch outlives the registrant. The registrant provably ran, so the defect is
between minting the verdict and the watch being retired — not in whether teardown reached the
registrant at all. - 🚨 Do not catalogue it in
.github/known-flakes.json. That would teachmerge-queue-steward
to re-queue past a real race. Flakes here are real races — this one has a production twin in its
own assertion text. - Control in both directions: the armed waiter receives a minted
OwnerDisposingverdict under
mesh teardown, AND an ordinary teardown with nothing owed still retires its watches promptly. A
fix that simply never retires a watch would pass the first and leak on every teardown. - Re-read OwnerDisposing NACK is dropped exactly when the parent is disposing hosted hubs — the caller hangs for its full budget #2778's original fix and say why it stopped holding — a recurrence whose first fix is
not understood tends to come back a third time.
Where it lives
The test is in MeshWeaver.Plugins (src/MeshWeaver.Hosting.Monolith.Test/NackReachesTheWaiterDuringTeardownTest.cs:236);
the code under test is core's hub teardown. Filed here because the defect is core's.
Related: #2778 (closed, the original) · #3510 / #4009 (the sibling seam, fixed) ·
#3965 · Doc/Architecture/DisposalStallVerdicts
🤖 Generated with Claude Code
Activity
DIAG #4023 v2: mirror a correlated reply's stages onto its request tr…
DIAG #4023 v2 fix: RequestFateLedger.RequestFate is nested — NEVER SHIP
DIAG #4023 v3: on FAIL snapshot the hub tree, time the watch's exit, …
NOTES #4023 (investigation state, 2026-09-11 ~14:08Z) — NEVER SHIP th…
NOTES #4023 update (~14:17Z) — NEVER SHIP this branch
fix(#4023): review — FIFO with the activation serializer, registered-…
rbuergi commentedon Sep 11, 2026
Status of the caller adoption (requirement 6), 2026-09-11 ~15:00Z.
Landed in core
pending:. This PR adds the Pluginspublish-modulesrow that way. It also addsnode-repo-module-pack.yml'sdeclaredoutput, andmodule-publication.py check-callers, whichnode-repo-validatenow runs on every satellite PR. The lane's caller-graph check therefore runs before merge instead of first onmain.unrelated-jobsexemption with an empty reason, refuse apublish-modethat isn't exactlystagedordirect, and make a spentpending:marker that has an empty reason fail. The docs are scoped to what actually ships.Durable write-up:
Doc/Architecture/ModulePublicationGate(caller, both directions,check-callers, landing order, not-solved list) andDoc/Architecture/WorkflowPermissionPairing(pending:).Open
modules-floortopublish-mode: stagedand addspublish-modules. Auto-merge is armed.Repo policy gates,Module bundles / All selected bundles built), and 2 were still running (Compile every NodeType (vs core),test-repos).Build + test the portal hostsis red on shard 3 only, the known red on Pluginsmainfrom core set 8345 (NodeTypeRecompileAlcLeakTest). Shards 0 to 2 pass. The fix isn't on Pluginsmainyet; Plugins#1656 is diagnostics only.origin/mainintofix/3878-staged-module-publication, runpython3 scripts/gen-manifests.py, and push. The new run then goes green and auto-merge lands it.fix/3878-roster-marker-spent(f41b0dc28), which removes the spent marker. Do not open it earlier: I checked that this roster against a Pluginsmainwithout the caller fails--assert-fleet-row(exit 1).publish-modules / Hand the validated bundles to the registrysucceeding, with registry 2xx lines for the staged bytes. A PR run exercises neither staging nor the hand-over (publishis false there).Not solved, and not claimed
plugins-bake.check-callersnames on every PR there.rn-app,e2e-static,test-drift,memex-template) gate neither publisher.rbuergi commentedon Sep 11, 2026
Fixed — #4036 merged (2026-09-11T15:18Z). Closing; I filed this, and the premise I filed it on was wrong.
Verified on
mainby file, not by PR state:src/MeshWeaver.Hosting/RoutingServiceBase.cscarries thechange, and
test/MeshWeaver.Graph.Test/ReplyRoutedDuringMeshTeardownReachesItsWaiterTest.csis present.The queue build ran all six test shards green on the merged tree.
The correction that matters
I titled this "#2778 has recurred" and asked why #2778's fix stopped holding. It never stopped holding.
(41/200 unstressed, 42/400 stressed).
owner.IsShuttingDown, the merge commits during teardown, the success ack claims the gate first and theregistrant stands down. No
OwnerDisposingNACK was minted in any failing run — the assertion's ownstory was false every time it failed. The catch-path elimination I recorded held, but it proved nothing
about a NACK.
The actual defect
RoutingServiceBase.RouteInMeshsilently dropped every message once the mesh reachedDisposeHostedHubs("recipients are likely also disposing") while still reportingForwarded— the"nobody is waiting" assumption at a third site. Failing trail:
PATCH_ACK ok→ response posted → meshreceives it while Quiescing → mesh routes it after its own
DisposeHostedHubsturn → never arrives,while the cache hub is Quiescing with exactly that callback pending. Monolith and CI only;
OrleansRoutingServicehas no such guard.Controls — each fails against exactly the variant it guards
Stress: 0 failures in 1,600 with the fix vs 4 in 3,600 before (supporting only — the deterministic test
is the proof). Teardown cost unchanged (189 s base vs 187 s branch).
Left open, deliberately
The test's own residual race — judging the ordinary route at the owner's
Dead— seen once in 3,600stressed runs, never in CI. Documented in
DisposalStallVerdicts("Resolved 2026-09-11"), not fixed.🤖 Generated with Claude Code
rbuergi commentedon Sep 24, 2026
New sighting, 2026-09-24T15:19Z, in the merge queue. Group build https://github.com/Systemorph/MeshWeaver/actions/runs/36019287652 (
gh-readonly-queue/main/pr-5664-…), jobRun tests (shard 4):MeshWeaver.Graph.Test.NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCaller—Did not expect collection {"LW1lBiz2VEaR2HYBc7KHdQ"} to contain "LW1lBiz2VEaR2HYBc7KHdQ" because the owner minted an OwnerDisposing NACK for this patch, and a caller was armed and waiting.The group was #5663 (docs only) with #5664 on top. #5664 changes only
Memex.Portal.ServiceDefaults, which onlyMemex.Portal.Shared.Testreferences, so neither PR can reach Graph.Test. The same test tree passed in the previous group build for #5664, https://github.com/Systemorph/MeshWeaver/actions/runs/36017948638 (success). The steward left #5664 dequeued asqueue-rejected.Not established: whether this is the #4023 mechanism or a new path to the same assertion. I did not read the shard's trace log. I have not reopened this issue; triage should decide that. (The triage inbox on the control instance was not reachable from this session.)