A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it

Measured: CD 7950 (`f55b9f4ab`, carries #3480 and #3475, neither #3501 nor #3505) failed its seal on a THIRD mechanism

[FAIL] Hosting — install: TimeoutException after the install's own 10-minute bound (── Hosting: installing 145 file(s)… at 23:36:36Z; the next package starts at 23:46:36Z). Every discriminator that names the two mechanisms fixed tonight reads zero over the whole bake log: VERDICT_TIMEOUT=0 (#3499/#3505), resubscribe failed=0 (#3498/#3501), LATE_NACK_REENQUEUE=0, OWNER_NACK_REENQUEUE=0, PATCH_FLUSH_FAULTED_SYNC=0. The control 7938 (sealed) reads zero on all of them too — so this run's failure is something else, and the trail says what.

The trail (request-fate ledger, printed by `[STALE-CALLBACK]`)

Four creates for Hosting's own nodes, all CreateOrUpdateNodeRequest → portal/nodeops-1SDY…, pending 30–60 s:

Hosting/Backup: … ROUTED onTarget=True state=Submitted@portal/nodeops(+5518ms) → HANDLER_ENTER → UPSERT_READ existing → update
  → UPSERT_WRITE_THROUGH_STREAM path=Hosting/Backup/_Activity/compile-state → HANDLER_EXIT state=Processed(+5519ms)
  ⇒ a handler was entered and no reply, completion or fault has been recorded since
(same for Hosting/BackupStore, Hosting/ConfigAudit, Hosting/Deployment)

and, in between, the owner of every one of those paths stops answering:

23:37:44  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/Admin/_Activity/compile-state       seq=338 bound=5000ms — the owner never acknowledged this write
23:37:44  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/ConfigAudit/_Activity/compile-state seq=339
23:37:44  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/BackupStore/_Activity/compile-state seq=340
23:37:51  [QUIESCE-TIMEOUT] Hosting: 1 callback(s) still pending after 2s — forcibly cancelling. Pending: CreateOrUpdateNodeRequest@portal/nodeops (… UPSERT_READ absent → create → HANDLER_EXIT ⇒ no reply recorded)
23:37:51  [PluginGating] reconcile failed for Hosting MeshWeaver.Messaging.HubDisposedBeforeResponseException: Hub Hosting was disposed before the response arrived
23:38:01  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/Backup/_Activity/compile-state seq=352 … Hosting/_Access/Public_Access seq=353 … Hosting/Deployment/_Activity/compile-state seq=355 … Hosting/LogEntry/_Activity/compile-state seq=366 … Hosting/BackupStore seq=369
23:38:25–23:38:54  [STALE-CALLBACK] Hosting/Backup, Hosting/BackupStore, Hosting/ConfigAudit, Hosting/Deployment — the four creates, 30…59 s and climbing

Reading

The Hosting root hub was disposed at 23:37:51Z while Hosting's own 145-file install was in flight. Its per-node children — the owners of Hosting/*/_Activity/compile-state, Hosting/_Access/Public_Access, Hosting/BackupStore — went with it. The nodeops handler had already written through those node streams and exited, owing its reply to the write's ack; the ack never came (ADVANCE_WITHOUT_HANDOFF, owner never acknowledged), so the reply was never sent, the installer's Observe never fired, and the install ran out its bound. This is the sequence #3499 was FILED as — "a package's ROOT hub is disposed mid-install" — and it is exactly what #3499's root-cause work set aside: the [QUIESCE-TIMEOUT] count peaking in the sealed run showed that a root recycling under a reconcile is usually benign, which is true, but the discriminator is not the count — it is whether the recycled root is the package currently installing. Here it is. #3505 (classification of a committed patch's durability fault) cannot reach it: these are creates through nodeops, not UpdateRemote patches, and no verdict was ever produced to classify. #3506 (a hub past [QUIESCE-OK] still taking intake) is not it either: RECEIVED runLevel=Started@Hosting — the root was healthy when it took the request and was torn down under it.

Who disposed the root is not in the log at this level — no [QUIESCE-START] Hosting, no dispose source line. The leading candidate is the NodeType rebind recycle (the package's own NodeTypes compile during its install; a rebind posts DisposeRequest to the root), i.e. #3499's hypothesis (2), which was never measured because (1) was found first. The next occurrence needs the dispose source at Information on the bake host, or [QUIESCE-START] naming the reason.

A second, distinct anomaly in the same log, recorded so it is not lost: HomeAssistant's create trail ends RECEIVED@portal/nodeops → ENQUEUED@portal/nodeops → ROUTED onTarget=False state=Forwarded@mesh(+2ms) — the nodeops hub took the delivery and forwarded it AWAY as not-on-target, 33–48 s pending, then resolved. A routing bounce, not the disposal shape; HomeAssistant installed.

Expected

An install's writes must not lose their owner to a recycle of the very root being installed: either the recycle is deferred while an install holds the root (the install is the writer that should own the root's lifetime), or the recycle answers every in-flight write under it with a NACK the nodeops handler turns into the installer's reply, so the install fails in milliseconds with a name instead of running out a 10-minute bound. And [QUIESCE-START] on a root should name who asked.

Consequence for #3355

Live in 7950 with #3480 in, absent by count in 7938; base rate on this exact shape tonight: 1 run (7937 and #1422 were #3499's, 7941 was #3498's). By the transitive rule it joins the blocker set; the candidate 7960 (155b4fa17) can meet it. Found reading 7950's bake log with seal-read.sh.

Attribution: the disposer is `PackageInstaller.SettleRetypedRoot`, and the recycle is not the defect

Measured on core CD 7950 (f55b9f4ab, #3480 in, neither #3498 nor #3499), bake job
101577531852. Every elimination
below is made from code, not from a line's absence.

It is not #3499, #3498 or #3506

counter in 7950	value
`VERDICT_TIMEOUT`	0
`PATCH_FLUSH_FAULTED_SYNC`	0
`LATE_NACK_REENQUEUE` / `OWNER_NACK_REENQUEUE`	0
`JsonSynchronizationStream … resubscribe failed`	0
`RECEIVED runLevel=…@Hosting` at intake	`Started`

#3505 classifies a verdict; here none is produced at all. #3506's intake window is excluded by
the trail: the root was healthy when it accepted the request.

Who disposed `Hosting`

PackageInstaller.SettleRetypedRoot — the installer's own, deliberate, ordered recycle of the
root of the package it is installing. Code-proven:

PackageInstaller.cs:2945 — retypedRoot = placeholderRoot is not null ? root?.Path : null, and
placeholderRoot (:2500) is non-null iff !rootTypeIsStatic, i.e. iff
FindStaticNode(root.NodeType) is null.
A plugin package root is Store/Plugin, and the code states it is dynamic in two places:
PackageInstaller.cs:92 — "partition ROOTS whose node types are dynamic (e.g. Store/Plugin)" —
and PartitionTeardownCoverageGate.cs:89 — "declared in mesh content — Store/Plugin — that no
src/-side registration covers".
⇒ SettleRetypedRoot fires on every plugin package install. :2994 places it immediately
after adoption + the first release wave, and the comment beside it names this exact hazard:
"pointed a whole package's compiles at a hub this very method was about to dispose (PackageInstaller launches every NodeType compile fire-and-forget, then deliberately disposes the root hub those compiles are reading #1732)."

The timing fingerprint — 4/4 packages, disposal always inside the package's own install, always after its own adoption

package	`installing N file(s)…`	last adoption for its bundle	root disposed	lag
HomeAssistant	23:36:24.9	23:36:33.5	callback awaited 23:36:43.1	+9.6 s
Hosting	23:36:36.1	23:37:43.9	23:37:51.8	+7.9 s
Cornerstone	23:47:42.1	23:48:22.7	23:48:37.6	+14.9 s
RemoteControl	23:51:38.2	23:52:04.2	23:52:05.9	+1.7 s

Never before the adoption; once per package. No other candidate predicts that shape.

🚨 The recycle SUCCEEDED, and is not the defect

Zero [PackageInstaller] root … never answered after its recycle warnings — the root came back
inside RootReadyTimeout (20 s) every time.
The same fingerprint (reconcile failed for <pkg>: Hub <pkg> was disposed before the response arrived) appears 7–12× per bake in every run examined, including the sealed control #7938
(12 occurrences, sealed clean).

So "the root of the package being installed is disposed under its own install" is by design and
survivable. What is not survivable is the work in flight across it never being answered.

Eliminations, each from code

candidate	why not
stale-build convergence (`NodeTypeEnrichmentHelpers`)	`AutoRecycleOnStaleBuild` reads `Modules:AutoRecycleOnStaleBuild`; the tester never sets it ⇒ `false` ⇒ the watcher offers, never disposes
overlay self-heal	requires a fault card; all types compiled `Ok`
`HubRecycleExtensions.RecycleNode`	exactly one non-test caller in core — `RecycleLayoutArea`, a circuit-only GUI confirm flow; no circuit in a bake
MeshWeaver.Plugins / `PluginGating`	the repo posts no `DisposeRequest` outside tests and `MeshPlugin.cs`'s MCP tool — the disposer is in core
`NodeTypeRebindWatcher`	excluded by ORDERING, not by a log line: it fires on the root's retype, which is in the write stage — before the release waves — while every observed disposal is after adoption

The actual defect — a totality hole on the UPSERT lane

MeshExtensions.cs:~4740 / ~4930 subscribe to the inner CreateNode / UpdateNode with
onNext / onError only. When the owning root is disposed under the request the inner observable
neither emits nor errors, so no terminal arm runs, no PostFail is posted and nothing is logged.
The trail ends exactly there:

HANDLER_ENTER → UPSERT_READ absent → create → HANDLER_EXIT state=Processed  ⇒ (nothing)

Confirmed by the denominator: zero [CreateOrUpdate] lines of any kind in the entire 7950 log —
silence, not a fault. Unlike the patch lane there is no WriteVerdictBound equivalent to convert
that silence into a verdict, which is exactly why every counter above reads zero. Same class as
#3033 / #3194, one lane over.

Attribution gap this measurement hit, and what closes it

SettleRetypedRoot logs nothing on the recycle path — only the decline
("not recycling root {Root}", Information). So the tester log-level change in #3505 names the
decline and [QUIESCE-START], but never the positive recycle. One LogInformation at the
DisposeRequest post closes it, and is part of the fix.

Fourth occurrence — CD 7967 (`4c89dc6b6`, the first run carrying #3512) — and the transcript now names the recycle itself

Tally on runs that started with #3480 in: 7950 failed, 7959 failed, 7962 hit it and sealed, 7967 failed — four occurrences, three failed seals. Same package every time: Hosting, 145 files. All counters zero again (VERDICT_TIMEOUT, both re-enqueue arms, PATCH_FLUSH_FAULTED_SYNC; resubscribe-failed 3, elsewhere), ADVANCE_WITHOUT_HANDOFF path=Hosting/… ×12, install bound at 10 min.

With #3512's line, the attribution that took a code read on 7950 is one grep:

03:44:13.836  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/Admin/_Activity/compile-state     ← first owner loss, BEFORE the recycle
03:44:19.082  Install: Hosting: adopted 15 prebuilt assembly(ies) for 15 installed type(s)
03:44:19.082  [PackageInstaller] recycling root Hosting now that its in-package NodeType has a loadable build —
              the hub re-activates against the package's own configuration.
              Work in flight beneath this root is answered by the teardown, not abandoned.      ← #3512's line
03:44:21.859  [UpdateQueue] ADVANCE_WITHOUT_HANDOFF path=Hosting/Backup/…                          ← and the rest, after it
03:44:29.688  [QUIESCE-TIMEOUT] Hosting: 1 callback(s) still pending after 2s — forcibly cancelling. Pending: CreateNodeRequest@portal/nodeops-…
…             [FAIL] Hosting — install: TimeoutException

Two things the labelled trail adds:

The recycle's own message states the contract the create leg breaks. "Work in flight beneath this root is answered by the teardown, not abandoned" is true for the update leg (RegisterOwnerDisposingNack → re-enqueue, measured green) and false for UPSERT_READ absent → create, where the pending CreateNodeRequest is what [QUIESCE-TIMEOUT] cancels 10.6 s later and nothing answers. The sentence is the specification; the create leg is the gap.
The first owner loss precedes the recycle by 5.2 s (Hosting/Admin/_Activity/compile-state at 03:44:13.8, recycle at 03:44:19.1). So at least one write under Hosting had already lost its owner before SettleRetypedRoot fired — a per-type rebind or release-wave teardown of that child, not the root recycle. Whoever drives the reproduction should park the seam for BOTH: the root recycle and a child owner going away mid-create.

Base rate now: 3 of 4 started runs failed on this; the one that passed (7962) had the same three un-terminated compile-state writes (its VERDICT_TIMEOUT lines) and the installer simply was not waiting on them. Whether an install fails is whether the abandoned create is one the installer awaits.

The seam to park is the CHILD owner, not the root — which is why two root-park attempts failed

Adding the piece that connects the 7967 trail to the reproduction attempts, because together they name the seam and separately neither does.

What was tried, and how it failed

While building this fix I drove the teardown twice and could not put a create behind a parked hub:

Parking the partition root's primary stream does not put the create behind it — the root's Version never advances after the post, so the create does not route through the parked hub at all. Both fences timed out at 36 s.

That read as "the harness cannot reach the seam". It was more specific than that.

What 7967's trail shows

03:44:13.8   ADVANCE_WITHOUT_HANDOFF  path=Hosting/Admin/_Activity/compile-state   ← FIRST
03:44:19.082 [PackageInstaller] recycling root Hosting now that its in-package
             NodeType has a loadable build …
03:44:29.7   [QUIESCE-TIMEOUT] Hosting … forcibly cancelling.
             Pending: CreateNodeRequest@portal/nodeops

🚨 The first ADVANCE_WITHOUT_HANDOFF is 5.2 seconds BEFORE the root recycle. A child owner went away mid-create while SettleRetypedRoot had not yet fired.

That is the explanation for the failed fences. The create routes to the child (Hosting/Admin/_Activity/compile-state), not to the root — so parking the root's stream parks something the create never traverses. The root recycle is a later, louder event that happens to be in the same window; it is not the seam.

What that means for the fix

The reproduction must park a per-type teardown UNDER the root, on the path the inner CreateNode actually traverses — not the partition root's primary stream. That is the concrete form of the earlier note that this needs "a park on the seam the inner CreateNode traverses, or extracting the inner-write subscription into a pure composition the way ArmPatchAckWatcher was for the patch lane." The second remains the better shape because it makes the totality drivable without a mesh at all.
🚨 The recycle message's own sentence is the contract, and the create leg is where it is false:

"Work in flight beneath this root is answered by the teardown, not abandoned."

On the update leg that is true — measured: RegisterOwnerDisposingNack mints OwnerDisposing, the writer re-enqueues, the caller gets success=True. On the create leg it is false: the inner observable neither emits nor errors, no terminal arm runs, no PostFail, no diagnostic. Worth quoting verbatim as the fix's test name — a contract stated in a log line and violated one lane over is exactly what a regression test should pin.
Scope, unchanged: the update leg is already total. Only the create leg (UPSERT_READ absent → create) is uncovered.

On the log line

It was added in #3512 for precisely this and named the recycle on its first run — 03:44:19.082, with adopted 15 prebuilt assembly(ies) immediately before it. Attributing the same event on 7950 previously required a full code read of PackageInstaller. That is the whole value of the line and it is now demonstrated rather than argued.

Occurrence count

Four occurrences, three failed seals, of four started runs since #3480 — Hosting every time. The one that passed (7962) hit the shape and survived; what separates them is presumably whether the un-terminated write is one the installer waits on. Hosting is the largest plugin package (145 files) with a dynamic root, which fits a window that scales with the install's own write count.

Fifth occurrence: CD 7968 (e64689e7f, the workflow_run twin of the commit 7969 sealed) failed its seal at 05:07:27Z on the same shape — [FAIL] Hosting — install: TimeoutException, all counters zero (VERDICT_TIMEOUT, both re-enqueue arms, PATCH_FLUSH_FAULTED_SYNC), resubscribe-failed 2 elsewhere. Tally over the eight runs that STARTED since #3480: sealed 7955, 7962, 7964, 7969; failed 7950, 7959, 7967, 7968 — every failure is this shape, Hosting every time, and 7962 carried the same un-terminated compile-state writes and sealed because the installer was not waiting on them. Roughly one started run in two loses its seal to this.

Sixth occurrence: CD 7976 (19b077868, workflow_run) failed its seal at 06:09:32Z on the same shape (Hosting install bound, all counters zero, resubscribe-failed 1 elsewhere). Tally over the ten runs that started since #3480: sealed 7955, 7962, 7964, 7969, 7974; failed 7950, 7959, 7967, 7968, 7976 — every failure this shape. One started run in two.

🚨 The create-leg attribution is REFUTED by the sixth occurrence — the park has no pending callback at all

Measured on CD 7976 (19b077868, bake job 101634597130), by reading the whole bake log rather than the counters.

The two facts that decide it

The one stale callback in the Hosting window took the UPDATE leg, and it resolved.

05:46:25.3 [STALE-CALLBACK] Hosting/InstanceRequest: 1 callback(s) pending > 30000ms:
  CreateOrUpdateNodeRequest@portal/nodeops(30089ms)
  … → HANDLER_ENTER → UPSERT_READ existing → update
    → UPSERT_WRITE_THROUGH_STREAM path=Hosting/InstanceRequest/_Activity/compile-state
    → HANDLER_EXIT state=Processed

Not UPSERT_READ absent → create. And it was reported once: the reporter re-reports every 5 s, so it was answered between 30 s and 35 s.

During the eight-minute park that actually ran out the bound, NOTHING was pending. [STALE-CALLBACK] fired 4× for HomeAssistant minutes earlier in the same run and 0× across the park. Every [STALE-CALLBACK] in the entire log: HomeAssistant ×4, WebSearch ×2, Training ×1, Hosting/InstanceRequest ×1 — none of them inside the park.

An unanswered CreateOrUpdateNodeRequest is a pending callback. The install was therefore not waiting on a node op; it was parked on an observable that never terminated. The create-leg totality gap is real — it just is not what loses these seals.

Where it does park

05:44:31.5  ── Hosting: installing 145 file(s)…
05:45:28.8  Installed node-repo plugin Hosting: 144 written, 0 unchanged
05:45:56.7  Install: Hosting: adopted 15 prebuilt assembly(ies) for 15 installed type(s)
05:45:56.7  [PackageInstaller] recycling root Hosting …          ← by design; it SUCCEEDS
05:45:59.6  InitializeHubRequest | Hub: Hosting                   ← the root is back, 2.9 s later
05:46:04.5  ADVANCE_WITHOUT_HANDOFF path=Hosting/Admin … /Backup … /LogEntry   ← the deferred wave
05:46:34.9  (last log line of any kind, at any level)
                                   ⋯ EIGHT MINUTES OF SILENCE ⋯
05:54:31.5  ── Hosting.Instance: installing 3 file(s)…            ← exactly T+600 s

The control is in the same log. Edu — same shape, same run — printed [PackageInstaller] warmed installed root Edu 2.5 s after its own deferred release wave. warmed installed root Hosting never printed at all, and WarmInstalledRoots runs immediately after RequestReleases(deferredWave) — so the park is inside the wave.

Why the wave and not something else in that segment, eliminated from code: AffectedNodeTypes carries TypeEnumerationBudget + a Catch; SeedPrebuiltAssemblies carries SeedBound; the trigger write carries BaseStateWaitBound (30 s) then verdict windows of LateResponseWatchBound + VerdictBoundGrace (31 s) across MaxConflictRetries. nodeTypePaths.Select(ObserveNodeTypeRelease).Merge().ToList() carries none — which MeshNodeStreamHandle.BaseStateSource's own remarks named in advance: "no per-leg bound and no outer bound — so ONE non-terminating leg parks the entire package install, silently, until the gate's own 600 s InstallTimeout reports install: TimeoutException against a package that installed fine 8 minutes earlier."

The defect, and the fix

ObserveNodeTypeRelease promises exactly one emission, always (its remarks, and its closing DefaultIfEmpty(false)). It delivered that for three of Rx's four outcomes — DefaultIfEmpty covers "completed empty", two Catches cover "faulted". Nothing covered "never terminated at all", which neither operator can observe.

#3550 adds BoundReleaseLeg: Take(1) first (so the deadline is TOTAL, not Rx's inter-emission one — the trap BaseStateSource was rewritten to remove), then ReleaseRequestBound, whose elapse answers false, logs at Warning and names the NodeType. 180 s: strictly above the ≈124 s a legitimate write can compose, far below the installer's 600 s. Nothing retried, nothing swallowed, no existing bound moved.

🚨 It does not claim to explain why a leg stops answering — that stays open, and the next occurrence will now name the type in one line instead of costing a full log read. Pinned by ReleaseWaveLegIsTotalTest (pure composition, TestScheduler, no mesh) — including the counterfactual: the unbounded wave over the same input still has not completed at 900 s.

Committed to Doc/Architecture/BakeSealNodeOpsSaturation and corrected in Doc/Architecture/WriteVerdictTotality.

Landed — #3550, merge commit `eb2b8d76a` (08:11:46Z)

Deliberately leaving this issue open, because what merged is not what this issue names.

What is fixed: the install no longer parks. ObserveNodeTypeRelease now keeps the contract it
already stated about itself — exactly one emission, always — for all four of Rx's outcomes, so one
non-terminating leg can no longer hold Merge().ToList() and, with it, the whole package install
until the gate's 600 s bound. A leg that does not answer within ReleaseRequestBound (180 s, ordered
above the ≈124 s a legitimate trigger write can compose and far below the installer's 600 s) is
answered false, logged at Warning, and names the NodeType.

What is NOT fixed, and is what should keep this issue open: why a release leg stops answering.
The eight-minute park in 7976 had no pending callback and produced no log line, so there was nothing
in the transcript to attribute it with. There is now.

On the next occurrence, this is the whole triage:

grep "RequestNodeTypeRelease.*produced no answer within" <bake job log>

One line, naming the NodeType path. That turns the residual from "read the whole bake log and
eliminate from code" into a single grep — which is the point of landing this half first.

Also corrected: Doc/Architecture/WriteVerdictTotality's #3510 section. The upsert lane's CREATE
leg gap is real and worth closing on its own merits, but it is not what fails these seals; 7976's
only stale callback took the UPDATE leg and resolved inside 35 s. Worth a separate issue rather than
staying filed under this one.

The create leg is now costing the fleet's critical path — a live sighting on Plugins #1463

This issue's remaining half (the upsert's create leg has no disposal-NACK) has just wedged the
MeshWeaver.Plugins pin move, which is currently blocking six PRs and five issues.

The sighting (measured by meshweaver-64 on Plugins run 34115469651, job 101732362683, gate
shard 1, tester log gate-log-8050550ab…-shard-1):

12:01:57  Installed node-repo plugin Hosting: 144 written
12:01:59–12:02:27  bake-seed adopts Hosting.zip 15/15 for identity s3a2556b…
          [STALE-CALLBACK] Hosting/InstanceAction:
              CreateOrUpdateNodeRequest@portal/nodeops-…  pending 30 → 58 s  "handler-side"
12:03:30  CompileStateMirror: satellite write failed,
              TimeoutException "No response received in hub Hosting/InstanceAction"
          … heartbeats only …
12:11:01  tester gives up on Hosting, starts Hosting.Instance   (compile budget expired)

Hosting never reached its ── Hosting: installed marker. Per-package pace is otherwise unchanged
against the green run (median 11.0 → 11.5 s; on 7938 Hosting took 108 s), so this is not a
systematic slowdown — it is one package hanging.

It matches this issue's own recorded fingerprint exactly. 7dd16fad0's measurement lists the
disposal timing per package — HomeAssistant +9.6s, Hosting +7.9s, Cornerstone +14.9s,
RemoteControl +1.7s — always inside the package's own install and always after its own bundle's
adoption. PackageInstaller.SettleRetypedRoot recycles the root it just retyped, and it fires for
every plugin package install (retypedRoot non-null iff !rootTypeIsStatic; Store/Plugin is
dynamic).

And core already says which leg. test/MeshWeaver.Graph.Test/UpsertAnswersWhenTheOwnerGoesAwayTest.cs:

:31 HANDLER_ENTER → UPSERT_READ absent → create → HANDLER_EXIT state=Processed ⇒ (nothing)
:51 "7950's trail is UPSERT_READ absent → create — the CREATE leg, which has no disposal-NACK"
:77 "the point of pinning it while A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510's create leg is still open"

That commit added 15 lines to PackageInstaller.cs, a doc, and the 177-line test pinning the
covered (update) leg. The create leg is untouched.

Shape. DispatchInnerCreate's hub.Observe(inner, …).Subscribe(…) has two arms — onNext and
onError. When the owner is disposed the response never arrives at all, so no arm ever runs. This
is the same class as #3550: a leg that can never terminate, not one that completes empty.
DefaultIfEmpty is the wrong repair, for the reason the read leg's own comment already gives — an
empty read there is turned into a create, which would be a silent write on missing information.

Ruled out, so nobody repeats the bisect: #3509, #3515, #3491, #3505, #3480. None of them is the
cause; the defect predates all five and 7991 merely made it legible.

_{Recorded 2026-09-07 during a fleet-wide sweep, from meshweaver-64's measurement plus a read of

core at the two pins (aa4075832 green, 67bd3d064 red).}

fix(#3510): the upsert's CREATE leg answers instead of hanging forever

Seventh occurrence — CD 8078 (`4fedf4549`, carries #3512, #3550 AND #3603) — a FOURTH arm: the UPDATE leg's post-commit flush parks under the root recycle

Measured on run 34190841613, bake job 101955088724, control = the sealed 8076 (34186868747, job 101941978797, same core commit, same identity s190dc0ca…). Full measurement and the code trace are on #3680 (the alert's auto-filed issue); the short form:

The gate red is idempotence: Chess — re-install of Chess/_Policy … TimeoutException — an UPDATE (PatchDataRequest), not a create, so fix(#3510): the upsert's CREATE leg answers instead of hanging forever #3603's create-leg NACK cannot reach it, and the trail shows no nack=OwnerDisposing: the per-node OWNER was never disposed. It stamped the patch (PATCH_MERGE_STAMPED v=4) and echoed it (PATCH_ECHO_SEEN +5 ms) and then recorded nothing — no PATCH_FLUSH_SUBSCRIBED, no bound armed, no ack, no VERDICT_EXPIRED, no "reached no route". The verdict was never minted.
Every one of the 15 such trails opens on [PackageInstaller] recycling root <X> now that its in-package NodeType has a loadable build (Hosting 06:11:45 → six Hosting/* writes; Chess 06:24:07 → seven Chess/* writes, the [QUIESCE-TIMEOUT] Chess … forcibly cancelling + HubDisposedBeforeResponseException in between). The control's single loss has the identical shape (recycling root Hosting 05:00:41 → FAILED Hosting/InstanceRequest/_Activity/compile-state 05:01:15).
Denominator: ~120 recycles per run; ~30 writes miss the 5 s handoff in ~6 roots in BOTH runs. The seal reds only when a stranded write is the idempotence re-install's own — this time the Chess recycle landed while Chess was being re-installed.
Mechanism (from code at 8b974323d): the flush is StoragePostCommitFlush.Build → WriteAndPublishUpdated → RoutingProxyAdapter.Write (.Take(1), no bound) — a routed write to the PARTITION hub whose prologue (_router.AddressFor, PersistenceService.TryWriteFrom's sequential Defer) runs on the owner's echo thread BEFORE the 10 s flush bound is armed (DataExtensions.cs ~1560–1590, the "Bake wedge: portal/nodeops stops draining mid-bake — one shared node-CRUD hub seen from eight places (queue latency bimodal: <=3.2s vs 33-49s) #2543 arm 1" window the code names). With the root mid-recycle the write has no live target, the prologue parks, no bound exists to fault it, and the caller times out at 31 s.

Taking this arm (meshweaver-64): deterministic repro first (a patch whose flush meets a recycling root), then the recycle drains or the flush faults — never the bounds. Cross-linked from #3680.

The disposer is named: it IS the installer's own retyped-root recycle

This issue's central open question — "Who disposed the root is not in the log at this level … The leading candidate is the NodeType rebind recycle … The next occurrence needs the dispose source at Information on the bake host, or [QUIESCE-START] naming the reason" — is now answered by measurement. #3603's "THE RECYCLE NAMES ITSELF" line landed 2026-09-07 14:16Z, and the next occurrence read it.

CD run 34190841613, 2026-09-08, package Chess (4fedf454, which carries #3603):

06:24:07.784 info  [PackageInstaller] recycling root Chess now that its in-package NodeType has a
                   loadable build — the hub re-activates against the package's own configuration.
                   Work in flight beneath this root is answered by the teardown, not abandoned.
06:24:08.936 info  [QUIESCE-START] Chess: 1 pending callbacks at dispose entry:
                   LIQeRW4kD0O3UIS2-JmXGw=CreateOrUpdateNodeRequest@portal/nodeops-…(1435ms)
06:24:10.939 warn  [QUIESCE-TIMEOUT] Chess: 1 callback(s) still pending after 2s — forcibly
                   cancelling. Pending: plDWeMJ1OU2E86iA4-Ixdg=CreateOrUpdateNodeRequest@…(241ms)
06:24:10.939 warn  [PluginGating] reconcile failed for Chess
                   HubDisposedBeforeResponseException: Hub Chess was disposed before the response
                   arrived (request type CreateOrUpdateNodeRequest, target portal/nodeops-…)

So it is PackageInstaller.SettleRetypedRoot — the installer's own retyped-root recycle — not the NodeType rebind watcher. That also falsifies #3499's stated reading, which set this candidate aside: "The Chess root is a Store/Plugin instance (PluginContent), so this is not the installer's retyped-root recycle." It is.

Two further details the trace settles:

The recycle's own claim is false as written. The line says "Work in flight beneath this root is answered by the teardown, not abandoned." The pending CreateOrUpdateNodeRequest was neither answered nor NACKed — it was forcibly cancelled at the 2 s quiesce bound and surfaced to its issuer as HubDisposedBeforeResponseException. That is precisely this issue's Expected, still unmet.
This occurrence is the intake-during-quiesce shape, which A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510's Hosting case explicitly was not. The request-fate ledger reads RECEIVED runLevel=**Quiescing**@Chess (the Hosting trace read runLevel=Started@Hosting, which is why A hub past [QUIESCE-OK] still accepts new correlated requests — the intake gate opens one phase too late #3506 was ruled out there). Two of the pending callbacks came from the root's own PluginGating reconcile, one in flight at dispose entry and one issued after quiescing began — so the root kept taking correlated work from its own background pipeline while being torn down under it.

There were two recycles of Chess per bake in the run that passed 65 minutes earlier (34186868747, 05:13:01Z and 05:13:50Z), each killing a reconcile pass — and the same double-recycle is visible on Video at 06:25:06.9 and 06:25:08.6, the second on an install that wrote 0 nodes (0 written, 21 unchanged). A no-op install still recycles the package root.

Scope note

This recycle race is not what made that run's seal go red, so fixing it would not have saved it. The red came from a second, independent defect — core's pre-#902 access heal fighting PluginGate.SeedGating over _Policy and the child denies until the heal's own write starved for 20 s — fixed in #3681, with the analysis in Doc/Architecture/PartitionAccessOwnership. The two compound (the recycle kills the reconcile pass, leaving more un-converged residue for the heal to find) but neither causes the other.

Recorded so the attribution is not lost. No bound was widened.

#3800 merged — the pointless half is gone; leaving this open for the half that has a job

What landed: SettleRetypedRoot now declines when the install wrote nothing, on the same condition as the two RequestReleases waves it sits between. That inconsistency was the bug — the waves know a no-op install has nothing to recompile, and the teardown between them did not.

Directly removes the two shapes measured on CD 34190841613: Video (0 written, 21 unchanged, recycled anyway) and the second of Chess's two recycles per bake.

And the positive log line stopped over-promising. It ended "Work in flight beneath this root is answered by the teardown, not abandoned." — measurably false in that same trace: the pending CreateOrUpdateNodeRequest was force-cancelled at the 2 s quiesce bound and reached its issuer as HubDisposedBeforeResponseException. It now says that.

What keeps this open

An install that did write still recycles a root whose background pipeline is issuing correlated requests. The thread's own reading names the deeper half — RECEIVED runLevel=Quiescing@Chess, i.e. the root kept taking correlated work from its own pipeline while being torn down under it. That is a hub accepting work past quiesce; its own defect, its own family, and not something to merge into a recycle-gating change. Widening the 2 s window would be the band-aid.

Pinned by RetypedRootRecycleNeedsAJobTest — 5 pure cases including the control (AnInstallThatWroteSomething_StillRecycles), because a predicate that declined every recycle would satisfy the regression case while silently re-breaking the binding the recycle exists to fix (#1732).

One of the three Expectations, in flight: #3821

"And [QUIESCE-START] on a root should name who asked."

#3821 does that. It matters more than its size because it is the Expectation that gates the
others: this issue could not identify the disposer at all, so "the leading candidate is the NodeType
rebind recycle … #3499's hypothesis (2), which was never measured" stayed unmeasured.

🚨 And the obvious source could not have supplied it. Dispose() posts its ShutdownRequest
to itself, so that message's Sender is always the dying hub — the field that looks like it
answers who asked is the one field that cannot. Anyone reaching for it would have got a
confidently wrong answer. The discriminating fact is one frame earlier, in HandleDispose: whether
a DisposeRequest arrived over the bus at all, and from where.

Three readings, and each rules something out:

line	means	rules out
`requested by portal/installer (routed DisposeRequest)`	another hub asked — `PackageInstaller` posting to a package root is exactly this issue's shape	a recycle; a host teardown
`requested by itself — a self-posted DisposeRequest (…), i.e. a rebind or self-heal recycle`	the automatic recycles, which post to their OWN hub	an external actor
`requested by a direct Dispose() (no routed DisposeRequest)`	host teardown, an owner disposing children, a `using`	the whole message path

The self-posted case is spelled out rather than printed as an address on purpose: NodeTypeRebindWatcher
and WithOverlaySelfHeal post to themselves, so a bare sender renders Hosting: requested by Hosting — true, useless, and easy to misread as a routing oddity. That is this issue's leading
hypothesis, so it is the one reading that must not be ambiguous.

What is still open here, and stays open

Defer a recycle while an install holds the root, or NACK every in-flight write under it
so the install fails in milliseconds with a name instead of running out a ten-minute bound.

Neither is touched. #3800 (merged today) removed a trigger — the retyped-root recycle no longer
runs when it has no job — and 3893dc486 (#3683) armed the patch-flush bound before the flush is
built so a root recycle cannot strand the verdict. Both reduce the occurrence; neither is the
contract this issue asks for. This issue should not be closed on #3821.

🚨 One caution for whoever picks up the rest, learned on MeshWeaver.Reinsurance#178 today: I expected
3893dc486 to be the transition there and it was not — three gate-shard runs went green on the
pre-fix binary. Occurrence counts moving is not evidence a specific commit moved them, especially
where the runtime is a floating tag.

#3832 is merged at 7f37c487bf63bffaa9b61d0c60e08608d63fcc0b, after successful PR and merge-queue CI. It fixes the separately measured own-outgoing intake hole: while Quiescing, a hub no longer treats its own new awaited request as third-party transit merely because its destination is elsewhere.

The deterministic regression failed before the fix because the destination answered normally. Coverage after the fix verifies immediate ShuttingDown refusal attributed to the origin, inbound refusal, existing reply drain, and third-party forwarding; the full messaging suite passed 405 tests without skips.

Leaving #3510 open. This closes admission of new owned requests during quiesce, not the remaining contract for creates already in flight across an install's root recycle. #3833 separately corrects the proven false NOT applied claim on an inner-create timeout and is still in the queue. No live rollout verification is claimed here.

Reopened — the 2026-09-10 sweep was not a triage decision

This issue was closed not_planned at 2026-09-10T14:31:18Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: Several arms landed; the Expectation this thread says gates the others has not.

Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.

docs(#3510): record the rebind elimination and where the reason is read

fix(#3510): the lease names the root the install WRITES under, and it…

The remaining half landed — #4009, merged `1c8354d9f`

"either the recycle is deferred while an install holds the root (the install is the writer that should own the root's lifetime), or the recycle answers every in-flight write under it with a NACK"

DEFER, and the other one is rejected on evidence. Faulting every in-flight write under a recycling root was measured wrong one lane over in #3112 — the caller's terminal has already fired at LOCAL_EMIT and the 5 s handoff bound is the writer's patience, not a statement about the store, so a fault there reports failure for a write that landed. The NACK arm that is correct — the owner ANSWERING with OwnerDisposing, which the writer re-enqueues — already exists on the update leg and is already green (UpsertAnswersWhenTheOwnerGoesAwayTest).

The rule

The install is the writer that owns the root's lifetime. While an install holds a package root, a recycle aimed at that root waits, and runs the moment the install releases it.

PackageRootInstallLeases (mesh-scoped singleton). PackageInstaller.Install / InstallNodeRepoDelta hold TargetPartitionOf(id, manifest) for the whole run; HubRecycleExtensions.RecycleNode and NodeTypeRebindWatcher consult it before posting their DisposeRequest, logging once when they defer (naming the holder) and once when they proceed.

"Released" always arrives

The lease is taken through Observable.Using, so it is released on OnCompleted, on OnError and on unsubscribe — the complete set of ways an Rx subscription can end. A failed install releases; an abandoned one releases; the mesh takes the registry with it. No timer force-releases a lease — "defer" has to mean wait for a state that always arrives, and the release arms are pinned by name.

Two things deliberately NOT deferred, both to avoid a deadlock

The subtree. An install waits on the rebuilds of the NodeTypes beneath its root, and the recyclers that serve those rebuilds live on the per-type hubs. The lease key is the whole root path, matched exactly.
SettleRetypedRoot. It is the holder — its teardown is the install's own ordered act, placed between the two RequestReleases waves by PackageInstaller launches every NodeType compile fire-and-forget, then deliberately disposes the root hub those compiles are reading #1732, and WaitForRootReady holds the install until the address answers again.

What the lease removes is the second, unordered teardown: NodeTypeRebindWatcher firing on the placeholder retype this very install performed (#3965 established by code that it can). And because its subscription is registered for disposal on the hub it would recycle, the installer's own recycle cancels it. One recycle per install, not two.

Controls, both directions

#3965 measured that a root recycling with no install of its own is the COMMON case, so every regression case is paired with the same composition holding no lease:

regression	control
`ARetypeOfARootAnInstallHolds_DoesNotRecycleUntilTheInstallReleasesIt`	`ARetypeWithNoInstallHoldingTheRoot_RecyclesAtOnce`
`WhileAnInstallHoldsTheRoot_TheRecycleWaitsAndThenRuns`	`WithNoInstallHoldingTheRoot_TheRecycleProceedsAtOnce`
`TheRootIsHeldWhileTheInstallRuns_AndReleasedWhenItFinishes`	`AnInstallThatFAILS_StillReleasesItsRoot`

With the three production edits neutralised and both suites rebuilt, exactly those three go red with their own assertion text and every control stays green.

Review found two more holes; both are named rather than papered over

MeshOperations.Recycle posts directly and is NOT gated — an operator recycling a package root mid-install can still strand that install. The class remarks and the doc list every poster outside the gate, because a guard whose reach is assumed reads as a guarantee it does not keep (the defect this issue already corrected once, in the installer's own recycle line).
Two packages targeting one partition can still interrupt each other. The obvious repair — have SettleRetypedRoot wait for holders that are not its own — is wrong: B's SettleRetypedRoot would symmetrically wait for A's lease, and two installs each holding and each waiting is a mutual deadlock no timeout may resolve. The sound repair is per-root install serialisation, a separate change.

Also fixed in review: the lease key now names the root the install WRITES under (InstallRootOf), because InstallCode falls back to the shared type partition while the install-record rule falls back to the package id — a lease on <id> for a package writing type/<id> reads exactly like a lease that is working.

What is still open here

This does not make an install immune to every interruption. Per-type recyclers beneath a root are deliberately not deferred, and SettleRetypedRoot still tears a root down while that root's background pipeline may be issuing correlated requests — the shape RetypedRootRecycleNeedsAJobTest records and #3800 narrowed. What is closed is the class this issue named: nobody but the install itself takes that root down while the install is writing under it.

Written up in Doc/Architecture/DisposalStallVerdicts — extending the page #3965 wrote rather than adding another.

Re-checked 2026-09-12 — #4009's gate is LIVE and measurably firing in production CD; the issue stays open on the half its own source code still names

Two questions were put to this thread today: does the 09-11 SIGSEGV/ALC wave supersede it, and does #4009
finish it. Answers: no and not quite — with numbers for both.

1 — The collectible-unload work does NOT reach this path (family resemblance is not coverage)

#4042/#4046/#4048/#4053 make teardown wait, reactively, for collectible AssemblyLoadContext unloads. That is
the assembly lifetime, and the wait is consumed in exactly three places, all of them tests:
test/MeshWeaver.Fixture/HubTestBase.cs:155,174, test/MeshWeaver.Hosting.Monolith.TestBase/MonolithMeshTestBase.cs:298,1496,
test/MeshWeaver.Fixture/CollectibleUnloadDrain.cs. In src/ the only references are the DI registration
(Threading/IoPoolServiceCollectionExtensions.cs:47) and CompilationCacheService.RetireInto.

PackageInstaller, HubRecycleExtensions, NodeTypeRebindWatcher and the node-ops write path subscribe to
none of it. Different owner (assembly lifetime vs hub lifetime), different teardown, no reach. Nothing in
that wave cancels in-flight work either — CollectibleUnloadDrain has no timer and reports RETAINED rather
than forcing — so the "teardown lets work finish" constraint is intact on both sides.

2 — #4009 is running in production CD, and the gate fires on every package

Measured on CD run 8399, bake job 103506680455,
2026-09-12 05:44–05:52Z, on image 4c99ec269 (= today's main):

	count
packages installed (each then re-installed for the idempotence check)	62
`[PackageInstaller] recycling root …` (SettleRetypedRoot, actually recycling)	62
`not recycling root …` (#3800's decline on the 0-written re-install)	62
`NodeType rebind: deferring the recycle of '<root>' — an install is writing under that root` (#4009)	62
`[QUIESCE-TIMEOUT] … forcibly cancelling`	0
`[QUIESCE-START] <root>: N pending callbacks`, N ≥ 1	0
`[STALE-CALLBACK]`	0
`[PluginGating] reconcile failed … HubDisposedBeforeResponseException`	0
`UPSERT_CREATE_NO_VERDICT` / `UPSERT_CREATE_COMPLETED_EMPTY`	0
`root … never answered after its recycle`	0
`[FAIL] … install: TimeoutException`	0

(The 62 HubDisposedBeforeResponseException strings in that log are all inside #3800's rewritten recycle
message, not exceptions.) [UpdateQueue] ADVANCE_WITHOUT_HANDOFF still appears 21× — consistent with the
09-08 measurement that ~30 writes miss the 5 s handoff in sealed and failed runs alike; it was never the
discriminator.

Denominator for the recurrence claim: CD runs 8101–8400 (300 runs, 2026-09-08T09:31Z → 2026-09-12T05:27Z)
contain 124 executions of Bake + publish NodeType assemblies to portal storage. 19 of those failed; all 19
logs were fetched and grepped — zero contain install: TimeoutException, and 15 of the 19 died in job setup with
856–943-line logs, i.e. those runs never started a mesh and prove nothing (the other 4 ran the full path,
13.8k–16k lines). So ≥109 executions actually walked the install path with zero occurrences of this shape. Last
recorded occurrence remains CD 7976, 2026-09-07T06:09Z — before #3603 (fb3eddf74, 09-07 14:20Z) made the create
leg total.

3 — One new observability note: 62 deferrals, 0 "the install released" lines

WaitWhileAnInstallHoldsIt announces in both directions, and only the defer half ever printed. That is
explained, not broken: NodeTypeRebindWatcher.cs:115 arms the watcher as
instanceHub.RegisterForDisposal(Arm(...)), so the installer's own SettleRetypedRoot recycle disposes the
watcher and cancels the deferred recycle — "one recycle per install, not two", exactly as #4009 states.

🚨 The consequence for the next reader: from the log alone, "cancelled by the installer's own recycle"
(intended) and "the lease was never released" (a leak) are indistinguishable — both are silence. A leaked lease
would present as a root that quietly never rebinds, with no line anywhere. If that is ever suspected, the reading
is PackageRootInstallLeases.HeldBy(<root>) on a live pod, not the bake log.

4 — Why it does not close yet: the source itself says half remains

src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:4518, on today's main, verbatim:

"…it does not remove the need for the owner-side disposal NACK the UPDATE leg gets from
RegisterOwnerDisposingNack, which is the remaining half of #3510."

Verified rather than taken on trust: MeshNodeErrorCode.OwnerDisposing is minted on the patch leg only —
DataExtensions.cs:908–925 (HandlePatchDataRequest, RunLevel > Started ⇒ NACK) and
RegisterOwnerDisposingNack at :1182, whose parameter is IMessageDelivery<PatchDataRequest>. The CREATE leg
has a total bound (InnerCreateVerdictBound = WriteVerdictBound + 5 s, MeshExtensions.cs:4520) and a
named refusal — so the 10-minute silent hang this issue was filed on is gone — but no owner-side NACK, so where
the update leg re-enqueues against the fresh activation and succeeds, the create leg answers
"the outcome is unknown" and the install fails.

And three disposer paths can still take a root while writes are under it, each named by #4009's own review:

PackageInstaller.SettleRetypedRoot — the measured disposer of CD 7950 and of CD 34190841613 — is
deliberately not gated (it is the lease holder). It ran 62× today; its log line now honestly says work in
flight "is given the quiesce window to finish; anything still pending at that bound is force-cancelled".
MeshOperations.Recycle posts directly and consults no lease.
Two packages targeting one partition can still interrupt each other — per-root install serialisation is a
separate change (the symmetric fix deadlocks).

Verdict — OPEN, narrowed to one sentence

The class this issue was filed on is closed and now measurably gated: nobody but the install itself takes that
root down while the install is writing under it. What keeps it open is the create leg's owner-side
OwnerDisposing NACK, which the code still annotates as this issue's remaining half, plus the three ungated
disposers above.

What the next occurrence must capture (so nobody re-reads a whole bake log): the trail stage after
UPSERT_READ absent → create — UPSERT_CREATE_NO_VERDICT (the 36 s bound fired: the owner went and nothing
NACKed) versus a nack=OwnerDisposing re-enqueue (the update leg's behaviour, which is what this half would
give the create leg) — together with the [QUIESCE-START] <root>: requested by … line naming the disposer, and
whether a deferring the recycle of '<root>' line preceded it.

_{Read-only: GitHub REST (no GraphQL), git show/git grep on origin/main, one bake-job log per failed run.

No mesh writes, no cluster access, no test runs.}

🤖 Generated with Claude Code

https://claude.ai/code/session_01KFBiMVHLTZM2axNgVvaZN2

2026-09-13 — unchanged; what #4154 does and does not reach here

Re-read on today's main: the remaining half is exactly as the 09-12 comment left it — MeshNodeErrorCode.OwnerDisposing is minted on the patch leg only (DataExtensions.cs HandlePatchDataRequest / RegisterOwnerDisposingNack), the create leg has its total bound and its named refusal and no owner-side NACK, and SettleRetypedRoot, MeshOperations.Recycle and two packages on one partition remain ungated. Nothing this session merged touches those.

Two things from #4154 do reach the shape this issue was filed on, and should be read at the next occurrence rather than the bake log alone: a handler that FAULTS on a package root past DisposeHostedHubs now answers its requester (arm 3 used to be dropped by ReportFailure's gate), and a NACK the root's parent cannot carry is handed to the requester in-process. Neither is the create leg's owner-side OwnerDisposing NACK — that is still the open half.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BRLdhAiX5DjdZC2WUfPg9E

2026-09-14 — the attribution arm is now COMPLETE; the create-leg NACK is the whole of what remains

Three things measured on today's main. Two of them close things this thread has been carrying; the third is unchanged and is now the only open item.

1 — Hole 2 of the 09-12 comment is CLOSED, and I ran the pin

That comment listed three ungated disposers. The second — "MeshOperations.Recycle posts directly and consults no lease" — no longer holds:

src/MeshWeaver.Mesh.Operations/MeshOperations.cs:3801 — return hub.WhenNoInstallHoldsRoot(resolvedPath), placed AFTER the permission fold and BEFORE RecycleCore, so the release-request STAMP (itself a write into the root the install is writing) is gated too, not merely the DisposeRequest.
Pinned from both entry points, deliberately not shared: OperatorRecycleWaitsForTheInstallHoldingTheRootTest (Memex.Portal.Shared.Test) and RecycleNodeWaitsForTheInstallHoldingTheRootTest (MeshWeaver.Graph.Test).
Measured, not assumed: 11/11 on the Memex.Portal.Shared.Test recycle/install filter and the Graph twin green, on a -c Release -warnaserror build of both projects.

So of the three: (1) SettleRetypedRoot is deliberately not gated (it IS the lease holder), (2) is gated, (3) two packages on one partition still needs per-root install serialisation — the symmetric fix deadlocks, as recorded.

2 — This issue's third Expectation is now satisfied END TO END, and the last anonymous disposer is gone

"And [QUIESCE-START] on a root should name who asked."

[QUIESCE-START] has named it since the disposeRequestedBy / disposeReason / cascadeOwner work. What was still missing is that the line that becomes an ISSUE is not that one. [QUIESCE-START] is Information; the red-log pipeline files Errors. #4316 (merged today, for #3712) therefore carries the same attribution onto the [DISPOSE-DISCARD] reports and into the NACK the stranded sender receives — the reader in this issue's own trail, who is in another process and sees none of the root's log.

And it removes the one hole in the attribution that nothing else could fill. Measured before the change: MeshOperations.Recycle was the only production poster of a DisposeRequest with no Reason, so an operator recycle printed "reason not stated by the caller" — and an operator recycle is precisely the teardown originating OUTSIDE the framework's own lifecycle, i.e. the one a log cannot reconstruct. After: grep -rn "new DisposeRequest()" src --include="*.cs" returns nothing; all 6 of 6 posters in src/ state a reason.

For CD 7950's actual question — "who disposed the root is not in the log at this level" — the next occurrence now answers it from the Error tier rather than needing Information-level Loki.

3 — What remains, and it is one sentence

The create leg's owner-side OwnerDisposing NACK. Unchanged and still annotated as such by the source itself — src/MeshWeaver.Mesh.Contract/MeshExtensions.cs:4595, verbatim:

"…it does not remove the need for the owner-side disposal NACK the UPDATE leg gets from RegisterOwnerDisposingNack, which is the remaining half of #3510."

The consequence is the one the 09-12 comment stated: where the update leg re-enqueues against the fresh activation and succeeds, the create leg has only its total bound (InnerCreateVerdictBound) and answers "the outcome is unknown" — so the install fails on a condition the update leg recovers from. The 10-minute silent hang this issue was filed on is gone; the asymmetry is not.

Not attempted here, and stated rather than left implicit: RegisterOwnerDisposingNack is ~100 lines whose every arm is load-bearing (eager DI capture out of the disposal path, the ILatePatchVerdictSink direct hand-off that replaced a parent-chain walk measured at ~10× on teardown, the once-only ack gate). Porting it to the create leg is a change of that size and deserves its own diff with its own repro, not a rider on a diagnostics PR.

What the next occurrence must capture (unchanged from 09-12, plus one field that now exists): the trail stage after UPSERT_READ absent → create — UPSERT_CREATE_NO_VERDICT versus a nack=OwnerDisposing re-enqueue — together with the [QUIESCE-START] <root>: requested by … line, which the [DISPOSE-DISCARD]/NACK path will now also carry.

_{Read-only for the measurements; grep on origin/main in a worktree; the two test filters run locally on -c Release -warnaserror builds. No cluster access, no mesh writes.}

🤖 Generated with Claude Code

Triage 2026-09-14: the remaining half is verbatim on main — and the gate #4009/#4202 built is only as wide as the writers that take a lease, of which there is ONE

MeshExtensions.cs:4595 still reads "…the remaining half of A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510"; RegisterOwnerDisposingNack (DataExtensions.cs:1182) is IMessageDelivery<PatchDataRequest>-only; OwnerDisposing is minted only on the update/patch lane; HandleCreateNodeRequest (MeshExtensions.cs:576-1174) has the Rx-completion backstop and no RegisterForDisposal NACK at all. Reader trap: merge db62521977 sits on branch fix/3510-create-leg-owner-disposing-nack but its diff is the MeshOperations.Recycle gate — the create-leg NACK has not landed.
🆕 grep -rn "HoldDuring\|leases\.Hold" src → one production site: MeshWeaver.PluginCatalog/PackageInstaller.cs:195 (HoldRootDuringInstall). Every other writer that lands a package tree holds no lease, so NodeTypeRebindWatcher.WaitWhileAnInstallHoldsIt and Recycle's WhenNoInstallHoldsRoot find nobody and proceed: (1) the GitSync import UpdateToLatestFromGitHub (MeshWeaver.GitSync/GitHubActivityExtensions.cs:165; zero PackageRootInstallLeases references in that project; driven unattended as System from ModuleDiscoveryService.cs:752), and (2) the in-mesh provision pipeline in Plugins Store/Publishing/Source/SystemInstall.cs — CreateRoot → RetypeRoot (a CreateOrUpdateNode retyping the root to Store/Plugin, exactly what RequiresRebind fires on) → WireSync → ImportLatest → CompileAllTypes, all unleased (git grep PackageRootInstallLeases origin/main in Plugins → 0). Established from code, not from a measured occurrence on those paths.

Fix sketch: port the owner-side NACK to the create lane (register it from HandleCreateNodeRequest on its existing once-only responded gate, eager DI capture, direct verdict hand-off) and re-enqueue the upsert's create arm on OwnerDisposing capped at MaxOwnerDisposingReenqueues; separately and cheaply, take PackageRootInstallLeases.HoldDuring around UpdateToLatestFromGitHub and around the Plugins SystemInstall.Run plan so the existing gates actually see those writers.

review(#4391): the fourth import entry point, `Ignored` is a dropped …

The remaining half landed — #4391, merged `4108aa463f` (2026-09-15 09:28Z)

"…it does not remove the need for the owner-side disposal NACK the UPDATE leg gets from
RegisterOwnerDisposingNack, which is the remaining half of #3510."
— MeshExtensions.cs:4595, on main until today.

That sentence is gone, and so is the thing it named.

Root cause, restated exactly

HandleCreateNodeRequest returns Processed() and owes its reply from a DETACHED chain whose legs
reach the partition bootstrap, the validators, storage and the post-creation handlers. A hub disposed
under any of those produces Rx's fourth outcome on that leg — never emits, never completes, never
faults — which neither DefaultIfEmpty nor Catch nor an onCompleted arm can observe. CD 7950's
trail ends there:

HANDLER_ENTER → UPSERT_READ absent → create → HANDLER_EXIT state=Processed  ⇒ (nothing)

#3603's InnerCreateVerdictBound made that a named refusal instead of a hang — but a bound expiring
is a statement about the caller's patience, not about the write, and it takes 36 s to make it.

What landed

1. RegisterCreateOwnerDisposingNack — the counterpart of
DataExtensions.RegisterOwnerDisposingNack, registered at handler entry and claiming the handler's
own once-only response gate. Its load-bearing rules are inherited: DI captured on the handler's TURN
(never resolved from the disposal path, which can run past a closed lifetime scope); the verdict
travels through the immediate parent only (walking the chain was measured at ~10× on teardown and
reverted); and no ILatePatchVerdictSink analogue, because that registry serves UpdateRemote's
armed write watches while a create's waiter is an ordinary hub.Observe callback that stays armed.

2. Claim-then-VERIFY on the verdict post. responded is now an interlocked claim, and a refused
post falls through to the parent — claiming the gate is exactly what disables the NACK (#3196, one
lane over). "Refused" includes MessageDeliveryState.Ignored, not only Failed: the storm breaker
returns it without enqueueing.

3. CreateNodeResponse.NodeError, and it survives the trip to an exception —
NodeCreationFailure.ToException stamps the whole MeshNodeError under NodeErrorKey, and the NACK
names the disposing hub in its own sentence. The requester is usually in another process and sees
none of the owner's log.

4. The lease's reach. grep -rn "HoldDuring\|leases\.Hold" src found ONE production holder
(PackageInstaller), so #4009/#4202's gate could not see the GitSync import — which writes a whole
Space tree including the NodeType retypes RequiresRebind fires on. All four import entry
points now hold (UpdateToLatest, UpdateToProvenCommit, ReconcileAtProvenCommit, Reimport);
CommitToGitHub deliberately does not, since it writes no mesh nodes.

🚨 What is NOT in it, and why — a withdrawal on evidence

An automatic re-drive was written twice and is not shipped. Neither site can keep the promise:

the upsert's own arm is structurally unreachable. The inner create is posted to THIS hub's
address, so the only activation whose disposal can mint that NACK for it is this hub — and a hub
past DisposeHostedHubs cannot take a re-drive. Unreachable code that looks like a recovery reads
to the next person as a covered case.
MeshService.CreateNode was implemented and driven against the rig, and it does not work. Node
CRUD executes on portal/nodeops-{meshId}, a stream-ROUTED address that is unregistered with its
hub and is not re-materialised by posting to it: the re-post is answered
RouteMessage: NotFound … No node found at 'portal/nodeops-…'. The retry buys one extra round trip
before the identical failure — a bound-widening band-aid.

So the create leg gets a verdict, not a recovery. That is this issue's own Expected, verbatim:
"the recycle answers every in-flight write under it with a NACK … so the install fails in
milliseconds with a name instead of running out a 10-minute bound." Making that address re-form is a
separate change, and the measurement is recorded in
Doc/Architecture/WriteVerdictTotality
so nobody re-derives it.

Measured, and the test failed first

CreateAnswersWhenTheOwnerGoesAwayTest parks a real INodeValidator inside the create chain — the
seam the create actually traverses, after this thread's two recorded attempts to park a partition
root's primary stream failed because "the root's Version never advances after the post, so the
create does not route through the parked hub at all."

	before	after
the caller	emits nothing at all for the full 36 s bound	answered at the disposal
through `IMeshService.CreateNode`	same silence	132 ms, code on `Exception.Data`, hub named
the control (hub stays up)	green	green

Three cases, plus GitSyncImportHoldsTheSpaceRootTest (all three release arms, the no-registry
pass-through, and that a sibling Space is not held — the mis-keying #4009's own review caught).
Suites: MeshWeaver.Graph.Test 1642/1642, MeshWeaver.Documentation.Test 473/473,
MeshWeaver.Hosting.Test 718/718, on -c Release -warnaserror.

Review found three things, all valid; two were misses

ReimportFromGitHub — the settings tab's manual re-import — held no lease, and was missed because it
is the one import that does not go through TriggerAuthorizedAsSystem. I pattern-matched the
writers instead of enumerating them, which is precisely the defect the section I wrote names. The
Ignored delivery state was the second. The third asked what pinned the re-drive; the answer was
"nothing could", and it is withdrawn above.

What stays open, elsewhere

Systemorph/MeshWeaver.Plugins#1907 — the provision plan
(CreateRoot → RetypeRoot → WireSync → ImportLatest → CompileAllTypes) holds no lease
either. git grep PackageRootInstallLeases in that repo returns 0. Established from code, not from
a measured occurrence.
The residual fix(#3510): an install OWNS its package root — a recycle aimed at it waits #4009 recorded: two packages targeting one partition can still interrupt each other,
and the symmetric fix deadlocks. Per-root install serialisation is its own change.

Closing this one: every Expectation it states is now met — the disposer is named, the recycle is
deferred while an install holds the root, and the in-flight create is answered with a NACK instead of
running out a bound.

🤖 Generated with Claude Code

A package root is disposed while its own install is in flight (CD 7950): the creates' write-through streams lose their owner, no reply is ever sent, and the install runs out its bound — neither #3501 nor #3505 covers it #3510

Description

Measured: CD 7950 (f55b9f4ab, carries #3480 and #3475, neither #3501 nor #3505) failed its seal on a THIRD mechanism

The trail (request-fate ledger, printed by [STALE-CALLBACK])

Reading

Expected

Consequence for #3355

Activity

rbuergi commented on Sep 7, 2026

Attribution: the disposer is PackageInstaller.SettleRetypedRoot, and the recycle is not the defect

It is not #3499, #3498 or #3506

Who disposed Hosting

The timing fingerprint — 4/4 packages, disposal always inside the package's own install, always after its own adoption

🚨 The recycle SUCCEEDED, and is not the defect

Eliminations, each from code

The actual defect — a totality hole on the UPSERT lane

Attribution gap this measurement hit, and what closes it

rbuergi commented on Sep 7, 2026

Fourth occurrence — CD 7967 (4c89dc6b6, the first run carrying #3512) — and the transcript now names the recycle itself

rbuergi commented on Sep 7, 2026

The seam to park is the CHILD owner, not the root — which is why two root-park attempts failed

What was tried, and how it failed

What 7967's trail shows

What that means for the fix

On the log line

Occurrence count

rbuergi commented on Sep 7, 2026

rbuergi commented on Sep 7, 2026

rbuergi commented on Sep 7, 2026

🚨 The create-leg attribution is REFUTED by the sixth occurrence — the park has no pending callback at all

The two facts that decide it

Where it does park

The defect, and the fix

rbuergi commented on Sep 7, 2026

Landed — #3550, merge commit eb2b8d76a (08:11:46Z)

rbuergi commented on Sep 7, 2026

The create leg is now costing the fleet's critical path — a live sighting on Plugins #1463

rbuergi commented on Sep 8, 2026

Seventh occurrence — CD 8078 (4fedf4549, carries #3512, #3550 AND #3603) — a FOURTH arm: the UPDATE leg's post-commit flush parks under the root recycle

rbuergi commented on Sep 8, 2026

The disposer is named: it IS the installer's own retyped-root recycle

Scope note

rbuergi commented on Sep 9, 2026

#3800 merged — the pointless half is gone; leaving this open for the half that has a job

What keeps this open

rbuergi commented on Sep 9, 2026

One of the three Expectations, in flight: #3821

What is still open here, and stays open

rbuergi commented on Sep 9, 2026

rbuergi commented on Sep 10, 2026

Reopened — the 2026-09-10 sweep was not a triage decision

rbuergi commented on Sep 11, 2026

The remaining half landed — #4009, merged 1c8354d9f

The rule

"Released" always arrives

Two things deliberately NOT deferred, both to avoid a deadlock

Controls, both directions

Review found two more holes; both are named rather than papered over

What is still open here

rbuergi commented on Sep 12, 2026

Re-checked 2026-09-12 — #4009's gate is LIVE and measurably firing in production CD; the issue stays open on the half its own source code still names

1 — The collectible-unload work does NOT reach this path (family resemblance is not coverage)

2 — #4009 is running in production CD, and the gate fires on every package

3 — One new observability note: 62 deferrals, 0 "the install released" lines

4 — Why it does not close yet: the source itself says half remains

Verdict — OPEN, narrowed to one sentence

rbuergi commented on Sep 13, 2026

2026-09-13 — unchanged; what #4154 does and does not reach here

rbuergi commented on Sep 14, 2026

2026-09-14 — the attribution arm is now COMPLETE; the create-leg NACK is the whole of what remains

1 — Hole 2 of the 09-12 comment is CLOSED, and I ran the pin

2 — This issue's third Expectation is now satisfied END TO END, and the last anonymous disposer is gone

3 — What remains, and it is one sentence

rbuergi commented on Sep 14, 2026

Triage 2026-09-14: the remaining half is verbatim on main — and the gate #4009/#4202 built is only as wide as the writers that take a lease, of which there is ONE

rbuergi commented on Sep 15, 2026

The remaining half landed — #4391, merged 4108aa463f (2026-09-15 09:28Z)

Root cause, restated exactly

What landed

🚨 What is NOT in it, and why — a withdrawal on evidence

Measured: CD 7950 (`f55b9f4ab`, carries #3480 and #3475, neither #3501 nor #3505) failed its seal on a THIRD mechanism

The trail (request-fate ledger, printed by `[STALE-CALLBACK]`)

Attribution: the disposer is `PackageInstaller.SettleRetypedRoot`, and the recycle is not the defect

Who disposed `Hosting`

Fourth occurrence — CD 7967 (`4c89dc6b6`, the first run carrying #3512) — and the transcript now names the recycle itself

Landed — #3550, merge commit `eb2b8d76a` (08:11:46Z)

Seventh occurrence — CD 8078 (`4fedf4549`, carries #3512, #3550 AND #3603) — a FOURTH arm: the UPDATE leg's post-commit flush parks under the root recycle

The remaining half landed — #4009, merged `1c8354d9f`

The remaining half landed — #4391, merged `4108aa463f` (2026-09-15 09:28Z)