Hub disposal wedges before Quiescing: the action block never dequeues the buffered ShutdownRequest(Quiescing) #3593

Activity

1. rbuergi commented on Sep 7, 2026

   

   rbuergi

   on Sep 7, 2026

   Contributor

   More actions

   #3615 merged. It does not fix the wedge — it makes the wedge nameable. Leaving this open.

   Three of the four "facts" in this report were artefacts of the instrument

   All from MessageService.GetQueueSnapshot, which returned (mainQueue.Count, deferredQueue.Count, 0, gates.Count, !draining, …):

   1. 🚨 exec=0 measured nothing. It was a hard-coded literal — a leftover from the TPL-Dataflow
      era; there is no executionBlock any more. "No turn is executing on this hub" is not what that
      field said. A verification step that cannot fail.
   2. 🚨 deliveryActionCompleted=False meant draining == true — a drain IS nominally in flight.
      The name said the opposite of the datum, and this is the most load-bearing field in the report.
   3. 🚨 last progress: … → Started did not mean the hub reached Started after Dispose().
      runLevelChanged is a BehaviorSubject, so the watchdog's own subscription inside Dispose()
      immediately re-emits the CURRENT level as its "observing from here" mark. The simpler reading is
      that the hub was already Started when Dispose() subscribed — which removes the
      Starting→Started race from the candidate set entirely.

   And the verdict asserted a cause the snapshot cannot support: "the stall is in a hosted hub or a
   join it is waiting on", on a hub at RunLevel=Started that has not reached DisposeHostedHubs and
   is therefore waiting on no child and no join. The taxonomy had a hole — queue non-empty, drain
   latched, nothing dequeued — and that hole fell into the "stalled below" bucket, sending every reader
   to children that do not exist. That is why 47 reports were unactionable.

   Merged #3557 is inert here and was neither cause nor fix: RefusesIntake returns false when
   runLevel < Quiescing, and exempts ShutdownRequest/DisposeRequest unconditionally.

   What #3615 changed

   Execution → DrainsInFlight, a real reading; DeliveryCompleted → Draining, naming its
   own datum; a new turnsDequeued counter incremented at mainQueue.Dequeue() (the existing
   turnsCompleted increments in RunHandler's .Finally, so it counts HANDLER completions and is
   blind to a turn that never reached a handler); event 7316 DisposalPumpNeverDequeued; 7313 is
   now guarded so it can only be printed when the hub actually is below; 7317 is an explicit
   unknown rather than a wrong guess.

   Falsified both ways — with only the verdict change reverted, the test fails on the old sentence and
   passes on THE PUMP IS NOT TURNING; the other four verdicts still pass (6/6 across
   DisposalStallWatchdogTest + DisposalDeadlockDiagnosticsTest + the new test). The new test's lever
   is a PausableTaskScheduler via WithTaskScheduler — it asserts on the emitted verdict, not that
   disposal completes, because asserting completion would claim a fix this PR does not make.

   What remains — and the next discriminator

   The wedge itself. Two mechanisms remain, and they could not be discriminated with the old
   instrumentation, which is the finding:

   • M1 — ScheduleDrainOne uses Task.Factory.StartNew(…, turnScheduler) without
     TaskCreationOptions.PreferFairness, so from a ThreadPool thread it enqueues onto that thread's
     LOCAL LIFO work-stealing queue; HostedHubsCollection.DisposeHubsReactive disposes every child
     sequentially on one thread, so N children's drains pile into that one thread's local queue,
     reachable only by stealing. Reproduces every measured field on all N children at once.
   • M2 — a drain IS running but blocked synchronously before its handler; the codebase already
     carries a dotnet-stack note for that exact frame (Monitor.Enter_Slowpath ← GetHub ← RouteStreamMessage ← DrainOne).

   One reading settles it at the next occurrence: drainsInFlight=0 and a dequeue delta of 0 ⇒
   M1. drainsInFlight ≥ 1 and a delta > 0 ⇒ M2.

   🚨 Deliberately NOT done: no PreferFairness, no retry, no watchdog re-kick, no widened bound.
   Each would make the symptom rarer while the mechanism is still unmeasured — and the fix for M1 and
   the fix for M2 are in different places, so guessing has a 50% chance of leaving the real one live
   and invisible.

   One operational note, checked: nothing in src/, .github/ or deploy/ keys on the literal 7313
   sentence (only a doc comment), so the rename blinds no triage rule — the incident opener fingerprints
   the line generically, and the pump shape will simply file as its own fingerprint under 7316.

   Design recorded in Doc/Architecture/DisposalStallVerdicts — field by field, what the snapshot
   actually measures, the taxonomy and its hole, and the M1/M2 table.
2. rbuergi commented on Sep 9, 2026

   

   rbuergi

   on Sep 9, 2026

   Contributor

   More actions

   #3798 merged. Leaving this open — it closes a third mechanism, it does not fix the wedge.

   Same call #3615 made, and for the same reason: what merged is real but is provably not what the 47 sync/* reports were.

   What landed

   The pump verdict still asserted the one load-bearing thing — "the drain flag is latched (a drain is scheduled on this hub's TaskScheduler)" — and nothing measured that parenthesis. drainsInFlight == 0 says no drain body is running; it does not say one is queued. Two states read identically and have opposite owners:

   	M1 — the scheduler holds it	M3 — nothing is outstanding
   drainsInFlight	0	0
   drainsScheduled − drainsStarted	> 0	0
   owner	the TaskScheduler	MessageService

   M3 was reachable. draining = true is set inside the turn gate in KickDrain; the schedule happens outside it, and Terminal() re-schedules with the latch still held. A throw from Task.Factory.StartNew at either site left the flag set with nothing outstanding — and since KickDrain returns immediately whenever the flag is set, the pump froze for the life of the hub, silently, with this issue's exact fingerprint and a verdict blaming a scheduler that had never been asked. ScheduleDrainOne now releases the latch and reports; it does not fall back to another scheduler, because the turn scheduler is the hub's serialisation guarantee.

   Why it is not your incident

   • Task.Factory.StartNew on TaskScheduler.Default does not throw.
   • A hosted hub is built from a fresh MessageHubConfiguration (IServiceProvider.CreateMessageHub) — nothing copies a parent's TaskScheduler, which is exactly what WithTaskScheduler's contract asks for ("use the grain's scheduler ONLY for the root grain hub").

   So sync/* hubs run on the default pool scheduler, where M3 is unreachable and a dead activation scheduler cannot reach them either.

   What remains, and what now prints

   M1-by-pool-starvation or M2, unchanged. The difference is that the next occurrence names itself: the verdict picks its MECHANISM: clause from drainsInFlight and drainsAwaitingScheduler instead of asserting one, and Doc/Architecture/DisposalStallVerdicts carries the three-way table plus the note that M3 does not apply to a hosted hub.

   Still no PreferFairness, still no widened bound.
3. rbuergi

   closed this as not plannedon Sep 10, 2026
4. rbuergi

   reopened this on Sep 10, 2026
5. rbuergi commented on Sep 10, 2026

   

   rbuergi

   on Sep 10, 2026

   Contributor

   More actions

   Reopened — the 2026-09-10 sweep was not a triage decision

   This issue was closed not_planned at 2026-09-10T14:31:03Z by a bulk sweep that PATCHed every open issue in all 13 Systemorph repositories — 311 issues in twelve minutes, no comment on any of them — after a session was asked to "close all issues in all repos". Nothing on this thread was read, and no judgement was made about this issue.

   All 82 issues the sweep closed in the MeshWeaver fleet were reviewed afterwards. This one is unresolved: The author wrote "leaving this open" twice: #3615 and #3798 make the wedge nameable, they do not fix it.

   Sixteen of the 82 were left closed — their fix had actually landed, or the author had withdrawn them; those should have read completed, not not planned.
6. rbuergi

   added a commit that references this issue on Sep 10, 2026

   docs(#3432): the population is not #3593 — zero of 4,649 carry Dispos…

   ...

   c174083
7. rbuergi commented on Sep 11, 2026

   

   rbuergi

   on Sep 11, 2026

   Contributor

   More actions

   Re-measured 2026-09-11 06:26–06:41Z — still OPEN, and for the reason this thread predicted: the wedge has not recurred, so the discriminating reading is still unavailable

   The closing condition as this thread states it

   Both merged comments say the same thing twice: #3615 and #3798 make the wedge nameable; they do not fix it. What closes this is the wedge — and the stated route is the reading at the next occurrence:

   drainsInFlight=0 and a dequeue delta of 0 ⇒ M1. drainsInFlight ≥ 1 and a delta > 0 ⇒ M2.

   Instrument and numbers

   1 — has it recurred? The incident this issue was opened from, read on the control instance memex.systemorph.com (/api/version = 3.0.0+45306a33e9437f1b662450dc23c4dfa6d897fe41, read 06:26Z):

   get @Admin/_LogIncident/969aad76bce7790c
     occurrences   47            (unchanged; occurrencesAtLastComment = 47)
     firstSeen     2026-09-07T11:36:41.116Z
     lastSeen      2026-09-07T11:36:41.157Z
     node lastModified 2026-09-07T11:41:56Z
   

   2 — is the instrument that would have caught a recurrence alive? Yes, and measured rather than assumed: search 'namespace:Admin/_LogIncident content.namespace:memex-cloud sort:lastModified-desc' returns incidents from the same namespace this burst came from updated 2026-09-11T06:30:14Z — four minutes before the read. The scanner is ingesting memex-cloud today.

   3 — the rename does not hide a recurrence. #3615 moved the pump shape to its own event (7316 DisposalPumpNeverDequeued), so a recurrence files under a NEW fingerprint. MessageHub.cs is the logger, so it would file under the MessageHub: category. Denominator: 12 of 12 MessageHub:-category incidents readable in Admin/_LogIncident, non-truncated — none is a pump/teardown verdict, and the newest of the twelve is ROUTER_TRAFFIC at 2026-09-10T22:10Z. (Admin/_LogIncident holds more than 200 incidents in total — the 200 listing came back truncated: true — so the 12 is the category slice, not the store.)

   4 — is the naming code actually running on the portals? Verified by FILE/SYMBOL at the two commits the portals serve, not by ancestry:

   ref	MessageHub.cs : DisposalPumpNeverDequeued	MessageService.cs : DrainsAwaitingScheduler
   origin/main	2	3
   6231c4da… (memex.meshweaver.cloud)	2	3
   45306a33… (memex.systemorph.com)	2	3

   Verdict — OPEN

   Nothing about the wedge is fixed, and four days without a recurrence is not evidence that it is: this fingerprint fired exactly once, as one mass event, on one pod. What has changed since the last comment is only that the naming code is now demonstrably running on both live portals, so the M1/M2/M3 reading will be printed by the next occurrence rather than having to be reconstructed.

   What remains, named: the reading itself — drainsInFlight and drainsAwaitingScheduler at a 7316/7317 occurrence. Plus the gap the 09-09 comment already recorded and which is unchanged: the verdict fires only from the disposal stall detector, so a hub merely stalled while alive never prints it — which is exactly the shape #2543 and #3674 keep hitting on portal/nodeops.

   _{Measured read-only through the portal API; no cluster access. Instruments: get/search on memex.systemorph.com, git cat-file/git show at the two running commits.}
8. rbuergi commented on Sep 12, 2026

   

   rbuergi

   on Sep 12, 2026

   Contributor

   More actions

   Re-checked 2026-09-12 against the 48-hour teardown wave — still OPEN, and this time the code is provably untouched

   The 09-11 comment left this on "no recurrence, so the discriminating reading is still unavailable". The
   question today is different: a large body of teardown work landed on 09-11 (#4042 / #4046 / #4048 /
   #4053 / #4017 / #4029 / #4051, and MeshWeaver.Plugins#1680). Does any of it remove M1 or M2? It does
   not, and the check is a file list rather than an argument.

   1 — What landed, by file

   merge	files it changes
   #4042 744796d8d	Compiler.Pipeline/CompilationCacheService.cs, Compiler.Pipeline/JsonMemberAccessorCacheEviction.cs, Mesh.Contract/Threading/CollectibleContextUnloads.cs, Threading/IoPoolServiceCollectionExtensions.cs (+4 lines), 3 test files, 2 docs
   #4046 5a10696fe	test/MeshWeaver.Fixture/CollectibleUnloadDrain.cs, MonolithMeshTestBase.cs, 1 doc
   #4048 cb724b571	MonolithMeshTestBase.cs only
   #4053 f1bd8c295	MonolithMeshTestBase.cs, a new test, a csproj, 1 doc
   #4017 b128b804d / #4029 dfc429b4a	CompilationCacheService.cs, MeshNodeCompilationService.cs, KernelScriptReferences.cs, tests, docs
   #4051 a41c79742	Hosting/NodeTypeBatchBake.cs, a test, docs

   None of them touches MessageService.cs, MessageHub.cs or HostedHubsCollection.cs. Measured
   directly: git log --since=2026-09-11 origin/main -- <those three files> returns nothing. The newest
   commit on any of the three is 55ce6895b (2026-09-10 20:54, #3510's disposal-cause diag), so the code
   this issue is about is byte-identical to what the 09-11 comment measured.

   Two further readings, because "teardown work landed" invites both of the wrong ones:

   • 🚨 The reactive collectible-unload wait is not a production teardown change. In src/ the only
     consumers of CollectibleContextUnloads are its registration (IoPoolServiceCollectionExtensions.cs:47)
     and CompilationCacheService.RetireInto; the wait lives in test/ (HubTestBase.cs:155/174,
     MonolithMeshTestBase.cs:298/1496, CollectibleUnloadDrain.cs). No hub teardown path subscribes to it.
   • ✅ Nothing was made forcible. CollectibleUnloadDrain carries no timer and no cancellation: it ends
     on the signal, on a fault, or on rounds that free nothing — which it reports as RETAINED rather than
     waiting or forcing. So there is no "we now abort in-flight work" story to reject here.

   2 — The two live mechanisms, still exactly as #3615 left them

   	on origin/main today
   M1 — drain enqueued onto a ThreadPool thread's LIFO local queue	MessageService.cs:1326 — Task.Factory.StartNew(DrainOne, CancellationToken.None, TaskCreationOptions.DenyChildAttach, turnScheduler). PreferFairness count in the whole file: 0, at both commits the portals are running.
   M1's amplifier — N children's drains piling into one thread	HostedHubsCollection.cs:551 DisposeHubsReactive still calls h.Dispose() sequentially in the projection (:568), one thread for the whole wave.
   M2 — a drain running but blocked synchronously before its handler	untouched.
   M3 — latch held with nothing scheduled	fixed by #3798 (ScheduleDrainOne's catch releases the latch) and still fixed.

   3 — The instruments are running on both live portals (symbol check, not ancestry)

   portal	/api/version	MessageHub.cs : DisposalPumpNeverDequeued	MessageService.cs : DrainsAwaitingScheduler
   memex.systemorph.com	74d4c8527	2	3
   memex.meshweaver.cloud	4c99ec269 (= today's main)	2	3

   4 — Recurrence: zero — and which kind of zero

   • The incident has not moved. get @Admin/_LogIncident/969aad76bce7790c (control instance): occurrences 47,
     lastSeen 2026-09-07T11:36:41.157Z, node lastModified 2026-09-07T11:41:56Z — unchanged since it was filed.
   • A recurrence would file under a new fingerprint in the same category, and the category slice is complete.
     search 'namespace:Admin/_LogIncident content.category:MeshWeaver.Messaging.MessageHub' → 8 of 8,
     truncated: false; none of the eight is a 7316/7317 pump verdict (newest in the slice: ROUTER_TRAFFIC,
     2026-09-12T06:23Z).
   • The instrument is ingesting today, not merely present: incidents in Admin/_LogIncident were written at
     2026-09-12T06:29:08Z, three minutes before the read.
   • 🚨 Denominator caveat, stated rather than glossed: this store is read as rbuergi and is RLS-filtered; "8 of 8"
     is the readable MessageHub: slice, not a proof about every partition.
   • 🚨 And the path is still being walked, so this is not an absence over runs that never ran it. The
     simultaneous mass-disposal shape recurs daily on memex-cloud: at 2026-09-12T06:22:22.557–.783Z a dozen
     root hubs (Import, Chess ×4, AppleWeather ×2, Indexing, LossModelling, ReinsurancePractice, …)
     disposed inside ~300 ms on one pod — fingerprint d2249f800ffc2577, event 7301. One cascade, every watchdog
     armed at once, exactly the population shape of this issue's burst — and no 7316/7317 was printed by it. What
     the incident store cannot say is whether any of those cascades reproduced M1's precondition (a saturated
     pool thread whose local queue holds the children's drains); that is unobservable from outside, which is why
     the closing condition is still a reading and not a count.

   5 — Closing condition, unchanged

   At the next 7316/7317: drainsInFlight = 0 with a turnsDequeued delta of 0 ⇒ M1;
   drainsInFlight ≥ 1 with a delta > 0 ⇒ M2. drainsAwaitingScheduler > 0 separates "the scheduler
   holds it" from "nothing is outstanding". Nothing about the wedge is fixed; five days of silence on a
   fingerprint that fired exactly once, as one mass event, on one pod is not evidence that it is.

   Still deliberately NOT done: no PreferFairness, no re-kick, no widened bound — the fix for M1 and the fix
   for M2 live in different places.

   _{Read-only: git log/git show/git cat-file on origin/main and on the two running commits; get/search
   on memex.systemorph.com; curl /api/version on both portals. No cluster access, no mesh writes, no test runs.}

   🤖 Generated with Claude Code

   https://claude.ai/code/session_01KFBiMVHLTZM2axNgVvaZN2
9. rbuergi

   added a commit that references this issue on Sep 13, 2026

   feat: the pending-callback report prints the target hub's pump state

   ...

   a44948c
10. rbuergi commented on Sep 13, 2026

    

    rbuergi

    on Sep 13, 2026

    Contributor

    More actions

    The M1/M2 reading is now printed from a LIVE hub — #4154

    This thread's closing condition is a reading at the next 7316/7317 (drainsInFlight, turnsDequeued delta, drainsAwaitingScheduler), and the 09-09 gap it recorded is that the verdict fires only from the DISPOSAL stall detector — a hub merely stalled while alive never prints it, which is the shape #2543/#3674 keep hitting.

    #4154 closes that gap from the other side. Under each pending callback whose target is a hub in this process, the pending-callback report ([STALE-CALLBACK], [QUIESCE-TIMEOUT], the disposal snapshot) now prints the TARGET's pump: target pump now: <address> RunLevel=… turn=<message> running <ms> buffer=… deferred=… drainsInFlight=… awaitingScheduler=… draining=…. Read against this thread's table: draining=True drainsInFlight=0 awaitingScheduler=0 with a non-empty buffer ⇒ M1 (the latch held, the drain never ran); drainsInFlight=1 turn=… running N ms ⇒ M2 (a drain running, blocked before its handler — and the turn is named); awaitingScheduler>0 ⇒ the scheduler holds it. Measured need: on the Plugins gate's cut run of 2026-09-13 (#4141), 11 SubscribeRequests sat at depth=1 on NodeType hubs for 31+ s with no line naming what those hubs were doing.

    Nothing about the wedge itself is changed by this; it stays open on the reading.

    🤖 Generated with Claude Code

    https://claude.ai/code/session_01BRLdhAiX5DjdZC2WUfPg9E
11. rbuergi commented on Sep 13, 2026

    

    rbuergi

    on Sep 13, 2026

    Contributor

    More actions

    2026-09-13 (later) — the M1/M2/M3 reading is armed and verified PRINTING; the wedge is byte-identical; no occurrence yet

    Three things measured today, none of which changes the verdict: still OPEN.

    1 — The reading prints, on a run rather than in a diff

    #4154 (e3d8ec06f, 08:37:54Z) puts this thread's discriminator on the pending-callback report of a hub that is merely stalled, not only one that is disposing — the gap the 09-09 comment recorded. Verified on core run 34757233860, job 103723972397 Run tests (shard 2), 22 occurrences. Verbatim:

          target pump now: host/1 RunLevel=Started   idle buffer=0 deferred=0 drainsInFlight=0 awaitingScheduler=0 draining=False openGates=0
          target pump now: host/1 RunLevel=Quiescing idle buffer=0 deferred=0 drainsInFlight=0 awaitingScheduler=0 draining=False openGates=0
    

    Read against this thread's own table: draining=True drainsInFlight=0 awaitingScheduler=0 with a non-empty buffer ⇒ M1; drainsInFlight=1 turn=… running N ms ⇒ M2, with the turn NAMED; awaitingScheduler>0 ⇒ the scheduler holds the drain. Every field is on the line.

    2 — The wedge itself is untouched, by file rather than by argument

    On origin/main today:

    M1 — the drain enqueued onto the calling pool thread's LIFO local queue	MessageService.cs:1532 — Task.Factory.StartNew(DrainOne, CancellationToken.None, TaskCreationOptions.DenyChildAttach, turnScheduler). PreferFairness count in the whole file: 0
    M1's amplifier — N children's drains piling onto one thread	HostedHubsCollection.cs:568 — DisposeHubsReactive still calls h.Dispose() sequentially inside the projection
    M2	untouched
    M3	still fixed by #3798

    The three pump files have moved only for #4183 (75b13b165, a refused reply is reported to the requester) and #4178 (3c421609e, a hub disposing with its own deferred delivery logs Debug rather than Error) — neither touches whether a buffered message starts a turn.

    3 — Occurrence since the instrument shipped: none, and the denominator says which kind of none

    target pump now: prints only inside a pending-callback report, so a green run printing none is expected. Between 08:37Z and ~15:45Z the bake/gate lanes that would produce one ran as follows: core CD's newest bake (34763464708/103746159798, 740 KB, a real mesh run — 4× ADVANCE_WITHOUT_HANDOFF) finished green with [STALE-CALLBACK] 0; the Plugins gate (34764523308, 4 shards, real mesh runs) finished green; SocialMedia's one gate died at NodeType compile before any mesh started; and Reinsurance, Education, Manufacturing and Crm reached no gate at all (all post-merge attempts died at validate / Validate node repos or were cancelled). Each consumed core commit was checked by symbol, not ancestry — 87805faaa, c9cd1f1b7, e50e4f588 all carry target pump now ×1 and reply# ×1 — so the instrument was in what ran.

    I did not re-read Admin/_LogIncident this session; the 09-12 reading of 969aad76bce7790c (47 occurrences, lastSeen 2026-09-07) stands as last measured.

    Verdict

    OPEN. Nothing about the wedge is fixed and nothing deliberately-not-done has been done: no PreferFairness, no re-kick, no widened bound. What changed is only that the closing condition is now readable from a hub stalled while alive, and that the reading has been seen to print with all its fields — so the next 7316/7317, or the next stalled-but-alive pump on a bake, names M1 / M2 / "the scheduler holds it" instead of having to be reconstructed.

    _{Read-only: GitHub REST (no GraphQL), git show/git log/git merge-base on origin/main and on the three consumed commits.}

    🤖 Generated with Claude Code
12. rbuergi commented on Sep 13, 2026

    

    rbuergi

    on Sep 13, 2026

    Contributor

    More actions

    2026-09-13 (later still) — asked the two questions the thread had left open. Still OPEN, and the reading is still unobtainable — but one of the two is now settled by construction rather than by inspection.

    The 15:59Z comment left this on "the reading is armed and verified printing; no occurrence yet". Two things were worth checking since, and neither had been:

    1. Does anything merged today make the reading obtainable rather than merely printable?
    2. Did fix: a reply is an ANSWER at every tier, and a refused one is reported to the REQUESTER (#4170) #4183's RefusesIntake change alter the mechanism? (fix(messaging): close the teardown intake gate at the first instant of disposal #3557's version was ruled inert on 09-07; the code has moved since.)

    1 — RefusesIntake is UNREACHABLE from this issue's state, and that is now a structural answer

    #4170/#4183 (01a81f3c3d, merged as 75b13b1651) really does change RefusesIntake — 111 insertions in MessageService.cs. What it changes is the DisposeHostedHubs tier:

    -        if (runLevel >= MessageHubRunLevel.DisposeHostedHubs)
    -            return true;
    +        if (runLevel >= MessageHubRunLevel.DisposeHostedHubs)
    +            return runLevel >= MessageHubRunLevel.ShutDown
    +                   || !delivery.Properties.ContainsKey(PostOptions.RequestId);

    Every line of it sits inside the method's first guard:

    var runLevel = hub.RunLevel;
    if (runLevel < MessageHubRunLevel.Quiescing)
        return false;

    This issue's whole population is at RunLevel=Started. RefusesIntake returns false on that first line and never reaches the changed code. So the answer is not "I read the diff and it looked unrelated" — the diff is unreachable from Started, which is the state every one of the 47 reports carries. Same verdict as #3557, for a stronger reason.

    2 — The pump is byte-untouched since the instrument landed, by SYMBOL not by stat

    MessageService.cs (+165/−10) and MessageHub.cs (+171/−2) have both moved since e3d8ec06f (#4154), so a file-level "unchanged" would have been wrong. The relevant question is narrower — does a buffered message start a turn? — so the check is on the symbols that answer it:

    git diff e3d8ec06f origin/main -- src/MeshWeaver.Messaging.Hub/MessageService.cs \
      | grep -E "^[-+].*(KickDrain|ScheduleDrainOne|DrainOne|drainsScheduled|drainsStarted|drainsInFlight|draining|mainQueue|turnsDequeued)"
    

    Zero hunks. And HostedHubsCollection.cs has no diff at all in that range. On origin/main (783892a859) today:

    M1	MessageService.cs:1532 — Task.Factory.StartNew(DrainOne, CancellationToken.None, TaskCreationOptions.DenyChildAttach, turnScheduler). PreferFairness count in the whole file: 0
    M1's amplifier	HostedHubsCollection.cs:568 — h.Dispose() sequentially inside DisposeHubsReactive's projection
    M2	untouched
    M3	still fixed by #3798

    The commits that DID touch those files since #4154 — e57ebd9395 (#1140, the ROUTER_TRAFFIC report names its call site), 3c421609e6 (#4178), 01a81f3c3d/3bad2b7446 (#4170/#4183), efebc3332a (#4067/#4068), 636a8c556c (#4169) — are all intake, reporting or init-retirement. None is on the drain path.

    3 — Recurrence: still zero, with the instrument measured ALIVE at the moment of reading

    • get @Admin/_LogIncident/969aad76bce7790c on the control instance (memex.systemorph.com): occurrences 47, lastSeen 2026-09-07T11:36:41.157Z, node lastModified 2026-09-07T11:41:56Z, occurrencesAtLastComment 47 — unchanged since it was filed, six days.
    • A recurrence files under a NEW fingerprint in the same category (fix: the disposal stall verdict measures what it asserts (#3593) #3615 moved the pump shape to event 7316). search 'namespace:Admin/_LogIncident content.category:MeshWeaver.Messaging.MessageHub' → 8 of 8, truncated: false; none is a 7316/7317 pump verdict. Newest in the slice: ROUTER_TRAFFIC, 2026-09-12T21:15Z.
    • 🚨 And the scanner is ingesting right now, not merely present — incidents in Admin/_LogIncident were written at 2026-09-13T21:30:07Z, about five minutes before this read. So the six-day silence is an absence over a live instrument, not over a dead one.
    • 🚨 Denominator caveat, stated rather than glossed: this store is read as rbuergi and is RLS-filtered. "8 of 8" is the readable MessageHub: slice, not a proof about every partition.

    4 — Does #4154 make the reading OBTAINABLE? No — and it is worth being precise about why

    #4154 puts the M1/M2/M3 discriminator on the pending-callback report, which closes the 09-09 gap (a hub stalled while alive could not print it). That is real and it is verified printing. But it does not reach this fingerprint:

    • It cannot manufacture an occurrence. target pump now: prints only inside a pending-callback report; a green run prints none, which is expected and is not evidence.
    • This issue's shape may not have a pending callback at all. The 47 reports are sync/* SynchronizationStream hubs disposed en masse at RunLevel=Started. Whether any of them has a callback outstanding at that instant is not something the captured window says. The path that does cover this fingerprint is the disposal stall detector (7316/7317), and that was already armed by fix: the disposal stall verdict measures what it asserts (#3593) #3615/fix(#3593): a failed drain schedule must not latch the pump, and the verdict must measure its mechanism #3798 — nothing merged today widened it.

    So the closing condition is unchanged and unmet: drainsInFlight = 0 with a turnsDequeued delta of 0 ⇒ M1; drainsInFlight ≥ 1 with a delta > 0 ⇒ M2; drainsAwaitingScheduler > 0 ⇒ the scheduler holds it.

    Verdict — OPEN, and deliberately nothing shipped

    Still no PreferFairness, no re-kick, no widened bound, no watchdog. M1 and M2 have fixes in different places; shipping one on a 50 % guess leaves the other live and, because the symptom would get rarer, invisible. A fingerprint that fired exactly once, as one mass event, on one pod, six days ago is not evidence that it is fixed — and it is not evidence that it is gone either. What changed today is only that the two ways it could have become stale (a pump edit, an intake-gate edit) have both been measured and neither happened.

    _{Read-only: git diff/git show/git log on origin/main and e3d8ec06f; get/search on memex.systemorph.com. No GraphQL.}

    🤖 Generated with Claude Code
13. rbuergi commented on Sep 14, 2026

    

    rbuergi

    on Sep 14, 2026

    Contributor

    More actions

    2026-09-14 — re-measured. Still OPEN, still not reproduced, and the mechanism is byte-unchanged.

    The brief I was handed carried a warning worth recording: an earlier session reported "my hypothesis is falsified — the parked-turn case is caught earlier" just before being cut off. I did not re-walk what it eliminated; I started where this thread says to start — does the wedge still reproduce at all — and the answer is no, for a kind of "no" that has to be stated with its denominator.

    1 — The incident has not moved in seven days

    get @Admin/_LogIncident/969aad76bce7790c on the control instance (memex.systemorph.com):

    field	value
    occurrences	47
    firstSeen / lastSeen	2026-09-07T11:36:41.116Z / 11:36:41.157Z (41 ms apart — one mass event)
    node lastModified	2026-09-07T11:41:56Z
    node version	8
    occurrencesAtLastComment	47

    Unchanged since it was filed.

    2 — The positive control is stronger than "the scanner is alive" — it is a SIBLING incident written today

    A zero over a dead instrument is not evidence. On the same store, read in the same minute, Admin/_LogIncident/d2249f800ffc2577 (the sibling teardown fingerprint, #3712) reads lastModified 2026-09-14T09:03:09Z, version 249, lastSeen 2026-09-14T08:57:09.895Z, occurrences 364. So Admin/_LogIncident is ingesting right now, and the seven-day silence on 969aad76bce7790c is an absence over a live instrument.

    🚨 Denominator caveat, stated rather than glossed: this store is read as rbuergi and is RLS-filtered. That is one incident record read directly by path, not a sweep, so the caveat is about whether a recurrence under a different fingerprint in a partition I cannot read exists — not about this record.

    3 — Both mechanisms are byte-present on today's main

    	on main today
    M1 — the drain enqueued onto the calling pool thread's LIFO local queue	MessageService.cs:1580 — Task.Factory.StartNew(DrainOne, CancellationToken.None, TaskCreationOptions.DenyChildAttach, turnScheduler). PreferFairness count in the whole file: 0 (grep -c)
    M1's amplifier — N children's drains piling onto one thread	HostedHubsCollection.cs:568 — h.Dispose() still called sequentially inside DisposeHubsReactive's projection
    M2 — a drain running but blocked before its handler	untouched
    M3	still fixed by #3798

    4 — What I merged today does NOT reach this, and saying so is the point

    #4316 puts the disposal attribution (requested by …; why: …) onto the [DISPOSE-DISCARD] reports and their NACKs, for #3712. That is the deferred-delivery arm of the teardown contract. It does not touch the pump, does not reach the 7316/7317 verdict, and does not make this thread's reading obtainable. It is listed here only so the next reader does not mistake a neighbouring teardown commit for progress on this one.

    5 — Closing condition, unchanged and unmet

    At the next 7316/7317: drainsInFlight = 0 with a turnsDequeued delta of 0 ⇒ M1; drainsInFlight ≥ 1 with a delta > 0 ⇒ M2; drainsAwaitingScheduler > 0 ⇒ the scheduler holds it.

    Still deliberately NOT done, for the reason this thread has given three times: no PreferFairness, no re-kick, no widened bound, no watchdog. M1 and M2 have fixes in different places, and shipping one on a coin toss leaves the other live and rarer, i.e. invisible.

    _{Read-only: get on memex.systemorph.com; grep/grep -c on origin/main in a worktree. No GraphQL, no cluster access, no mesh writes.}

    🤖 Generated with Claude Code
14. rbuergi

    closed this as completedin #4379on Sep 15, 2026
15. systemorph-com commented on Sep 19, 2026

    

    systemorph-comBot

    on Sep 19, 2026

    Contributor

    More actions

    Reopened — this fault is happening again.

    Still occurring: 10 more since the last update (57 total).

    • Last seen: 2026-09-19 08:55:21Z
    • Pods: memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-6457bb9544-qpdjd

    Recent log lines

    2026-09-19 08:55:05Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d made no teardown progress for 00:00:08 (last progress: sync/rua9uGeAlUayWAt1otk3_A → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 2 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 77026ms)
          
    2026-09-19 08:55:12Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d made no teardown progress for 00:00:08 (last progress: sync/rua9uGeAlUayWAt1otk3_A → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 2 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 83478ms)
          
    2026-09-19 08:55:21Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d made no teardown progress for 00:00:08 (last progress: sync/rua9uGeAlUayWAt1otk3_A → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 2 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 93076ms)
          
    

    _{Re-addressed by the current identity function: this incident inherited 969aad76bce7790c (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}
16. systemorph-com

    reopened this on Sep 19, 2026
17. systemorph-com commented on Sep 19, 2026

    

    systemorph-comBot

    on Sep 19, 2026

    Contributor

    More actions

    Still occurring: 23 more since the last update (80 total).

    • Last seen: 2026-09-19 08:58:09Z
    • Pods: memex-portal-deployment-69956b6dbc-gj8m2, memex-portal-deployment-6457bb9544-qpdjd

    Recent log lines

    2026-09-19 08:58:01Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub mesh/b_IjjWwcEk2RsrobrUhppA made no teardown progress for 00:00:08 (last progress: christoph.hofstetter/_Install/Systemorph-MeshWeaver.Plugins → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 9 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub mesh/b_IjjWwcEk2RsrobrUhppA RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
            HostedHubs (1):
              Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 253055ms)
          
    2026-09-19 08:58:05Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d made no teardown progress for 00:00:08 (last progress: sync/rua9uGeAlUayWAt1otk3_A → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 2 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 256978ms)
          
    2026-09-19 08:58:09Z memex-portal-deployment-69956b6dbc-gj8m2 fail: MeshWeaver.Messaging.MessageHub[7313]
          DISPOSAL DEADLOCK DETECTED: Hub mesh/b_IjjWwcEk2RsrobrUhppA made no teardown progress for 00:00:08 (last progress: christoph.hofstetter/_Install/Systemorph-MeshWeaver.Plugins → Dead). RunLevel=DisposeHostedHubs, queue depth 0. No turn is executing on this hub and its pump has dequeued 9 turn(s) since Dispose(), and it has 1 hosted hub(s) below it, so the stall is in a hosted hub or a join it is waiting on — the diagnostics below name it. Disposal is NOT forced.
          Hub mesh/b_IjjWwcEk2RsrobrUhppA RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
          HostedHubs (1):
            Hub Hosting/FleetConsole/_Activity/compile-202609190838333269cd7d8644fc94b6eb5627469d42e494d RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
            HostedHubs (1):
              Hub sync/SOngKWFItUOzwSchcdrx1w RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False) Executing(ShutdownRequest, 261055ms)
          
    

    _{Re-addressed by the current identity function: this incident inherited 969aad76bce7790c (Systemorph/MeshWeaver.Plugins#1796). Those nodes are superseded and will not fold, file or comment again.}
18. rbuergi commented on Sep 19, 2026

    

    rbuergi

    on Sep 19, 2026

    Contributor

    More actions

    The discriminating reading finally arrived — and it is neither M1 nor M2. The titled defect is resolved; this issue is now carrying a different, third shape

    This thread has said the same thing five times: #3615 and #3798 make the wedge nameable, they do not
    fix it, and the closing condition is a reading at the next 7316/7317. The next occurrence has
    happened. It is not a 7316, it is not the titled shape, and it settles both halves.

    1 — The titled defect: M1 is fixed and the original fingerprint is frozen and superseded

    The title is "the action block never dequeues the buffered ShutdownRequest(Quiescing)", measured as
    RunLevel=Started, buffer=1, exec=0. On the control instance:

    get @Admin/_LogIncident/969aad76bce7790c
      occurrences    47                          (unchanged since filed)
      firstSeen/lastSeen  2026-09-07T11:36:41.116Z / .157Z   (41 ms apart — one mass event)
      status         Superseded
      supersededBy   23170794ce38a460
    

    Twelve days, zero recurrence — and this time that is not dormancy measured across a quiet period.
    memex-cloud rolled or restarted six times on 2026-09-19 (05:03Z, 05:50Z, 06:14Z, 06:35Z, 07:39Z,
    08:32Z). Teardown is the path that fires this, it was exercised hard all morning, and it produced
    occurrences of other shapes on the same instrument in the same minutes — so the silence on this
    fingerprint is an absence over a live instrument under the regime that provokes it. That is the control
    the #1540 family's reopens showed was missing, and here it is satisfied.

    And M1 — the mechanism this thread named — is fixed and on the running image.
    MessageService.ScheduleDrainOne now asks for TaskCreationOptions.PreferFairness
    (MessageService.cs:1658), so the drain goes to the ThreadPool's GLOBAL queue instead of the posting
    thread's LIFO local queue. git merge-base --is-ancestor 4c8b5815f2 c25f86ae85b7 ⇒ true (fix
    2026-09-15 19:02Z; the running image built 2026-09-19 07:32Z). The code comment states it plainly:
    "That was #3593's mechanism."

    🚨 The successor's own triage draft agrees, independently of me:

    "The 47 occurrences from 2026-09-07 folded into this fingerprint are the already-resolved #3593
    population (old instrumentation, pre-PreferFairness, superseded incident 969aad76bce7790c) and
    add no new signal."

    2 — What the 80 occurrences actually are

    The growth 47 → 80 is the re-addressing bot folding successor fingerprints onto this issue, not the
    titled shape recurring. 23170794ce38a460 holds 33 occurrences of a new shape, lastSeen 2026-09-19T08:58:09Z, and it is a three-level chain pinned by one leaf:

    Hub mesh/b_IjjWwcEk2RsrobrUhppA          RunLevel=DisposeHostedHubs  (pump dequeued 9 turns)
      Hub Hosting/FleetConsole/_Activity/compile-…  RunLevel=DisposeHostedHubs  (pump dequeued 2 turns)
        Hub sync/SOngKWFItUOzwSchcdrx1w      RunLevel=Quiescing
            Queue(buffer=0,deferred=0,drainsInFlight=0,openGates=0,draining=False)
            Executing(ShutdownRequest, 253055ms)
    

    Read against this thread's table: the pump turned (both ancestors dequeued turns; the leaf reached
    Quiescing), so it is not M1; drainsInFlight=0, so it is not M2; M3 stays fixed. The
    titled premise — "the action block never dequeues the buffered ShutdownRequest" — is falsified for
    this population: it was dequeued.

    3 — Three defects found, all in the instrument or the state machine, all fixed in #4924

    I traced it from the code rather than from more log-reading. Findings, each verified at the site:

    (a) The 7314 verdict asserts a cause that is unreachable at this phase — the same defect #3615 fixed
    for 7313 and left in its sibling. DisposeImpl → disposables.Dispose and messageService.Dispose
    are reached only from case MessageHubRunLevel.ShutDown: in HandleShutdownCore. The leaf is at
    Quiescing, so "a registered cleanup is BLOCKING inside DisposeImpl or messageService.Dispose … find
    the registrant" is false by construction — no registrant has run. The branch keyed on the message
    type alone, with no phase check. This cost a full investigation through every disposal registrant of a
    sync/* hub (a dozen, in SynchronizationStream / JsonSynchronizationStream; none reachable at
    Quiescing). 7314 is now phase-guarded, and 7318 states what is true below ShutDown: the finding
    is the phase transition, with the QuiesceTimeout × MaxQuiesceRearms ≈ 42 s ceiling to read 253 s
    against, and the [QUIESCE-*] lines that separate "the wait never completed" from "it did and the
    hand-over was lost".

    🚨 It had no test at all — grep -rn DisposalShutDownPhaseBlocked src/ test/ returns only the site
    that emits it. Checklist item 7 from AFailureReportAnswersItsOwnInstruction, collecting another
    victim: a path no test can reach is a path whose wording nobody checks.

    (b) The Quiescing→DisposeHostedHubs hand-over was the one phase advance with no guard. Its sibling
    PostShutDownPhase has always caught a failed Post and called SignalDisposalFaulted so waiters on
    DisposalCompleted are released. The Quiescing advance was a bare Post in a finally — and the
    catch a few lines above it exists precisely because a throw there "would wedge the dispose state
    machine at Quiescing forever (worse than the original hang)". The one statement that comment is
    about sat outside it. A lost Post leaves exactly the observed shape: Quiescing, buffer=0,
    drainsInFlight=0, draining=False, Disposal=Pending, no further line of its own, every ancestor
    blocked behind it. Now force-faults — which is the opposite of swallowing: the fault is reported and
    the waiters above are freed.

    (c) Executing(ShutdownRequest, 253055ms) described a turn no thread was in. The snapshot is
    self-contradictory under this file's own invariants, and that is the finding: a handler holding the
    block synchronously requires drainsInFlight >= 1 (a turn runs only inside DrainOne); a turn parked
    asynchronously leaves DrainLoop without clearing the latch, so it requires draining=True. Neither
    held. Cause: RunHandler set currentlyExecutingMessageType near its top and cleared it in a
    .Finally attached ~260 lines later, so a throw between the two left the field set with nothing to
    clear it (Observable.Defer makes such a throw a downstream OnError with the Finally never
    attached). The elapsed then reads as time since a stamp nobody cleared — and a verdict keys on it.
    ExecuteOnTarget now stamps through Observable.Using, which disposes its resource both on
    termination and when the observable factory throws, so no exit path can stamp without arming the clear.
    turnsCompleted deliberately stays in the Finally — it counts HANDLER completions and must stay
    blind to a turn that never reached one.

    This is the second time a field on this snapshot has described something it could not measure. The
    first was exec=0, a hard-coded literal, found by this same thread on 2026-09-07.

    4 — Honest limits

    • Nothing deliberately-not-done has been done: no PreferFairness added (it was already there), no
      re-kick, no retry, no watchdog, no widened bound.
    • I could not identify the statement that throws in (c)'s window, and I did not guess one. The fix
      closes the hole by construction; the orphan is measured (the contradictory snapshot) even though
      its throw site is not located.
    • No falsifiable before/after test exists for (a)–(c), and I will not claim one. Reaching
      RunLevel < ShutDown with CurrentMessage == ShutdownRequest needs either the orphan (now closed)
      or a blocking Quiescing turn, and neither has an injectable seam — which is precisely why 7314 was
      untested and its prose wrong. What I do have: all three are provable by reading the sites above, (a)
      and (b) change only which line is logged / whether a fault is reported, and the full messaging
      suite passes 483 tests, 0 failed with all three in place, which is the evidence (c)'s hot-path
      relocation needs — that it is behaviour-preserving.
    • Consolidation: Hub disposal wedges before Quiescing: the action block never dequeues the buffered ShutdownRequest(Quiescing) #3593 is not one root with Pooled I/O leaf ignores its cancellation token, letting silo teardown dispose the mesh over live work #2480, A route leg stuck at silo shutdown outlives the 30 s quiescence budget and strands its sender without a NACK #2833 or Deferred-delivery discard at hub disposal reports an empty gate list — the gate that held the message is never named #3712. Separately, A route leg stuck at silo shutdown outlives the 30 s quiescence budget and strands its sender without a NACK #2833 and Deferred-delivery discard at hub disposal reports an empty gate list — the gate that held the message is never named #3712
      turn out to be the same event seen from two ends — same delivery id AfJyOHs0WUu57CggZ5yWkg, same
      hub RiskTransfer, same pod, six seconds apart on 2026-09-08 — which is recorded on both of those.

    Verdict

    Open, but for a different reason than yesterday: the titled defect is resolved (M1 fixed, on the
    image, fingerprint frozen and superseded across six rolls), and this issue is now the ticket for the
    successor's third shape. The three fixes above remove the instrument defects that made that shape
    unreadable and close the one state-machine hole that produces exactly its signature. What closes this
    is the next occurrence of 23170794ce38a460 on an image carrying #4924: a 7318 naming the
    transition, or a 7314 that is now only printed where a registrant genuinely can block.

    Taxonomy, the read-order caveat and the 2026-09-19 sighting are written up in
    Doc/Architecture/DisposalStallVerdicts.

    _{Read-only for the measurements: get on Admin/_LogIncident/{969aad76bce7790c, 23170794ce38a460} on memex.systemorph.com; git merge-base --is-ancestor / grep / dotnet build / dotnet test in a worktree. No GraphQL, no cluster access, no mesh writes.}

    🤖 Generated with Claude Code
19. rbuergi commented on Sep 19, 2026

    

    rbuergi

    on Sep 19, 2026

    Contributor

    More actions

    Correction to the comment above: the PR is #4931, not #4924

    My previous comment cites #4924 twice — in "all fixed in #4924" and "the next occurrence … on an
    image carrying #4924". That number was written before the pull request existed and is wrong.

    • The three fixes are in fix(#3593): the stall verdict below ShutDown, the unguarded phase advance, and an orphanable Executing tracker #4931 — fix/3593-verdict-phase-and-quiesce-advance.
    • docs(logwatch): one log site holds TWO buckets, and a re-addressed incident freezes the old one #4924 is an unrelated docs PR by another session ("docs(logwatch): one log site holds TWO buckets,
      and a re-addressed incident freezes the old one"). Apologies for the stray cross-reference it will
      have picked up; nothing in it relates to this issue.

    So the closing condition reads: the next occurrence of incident 23170794ce38a460 on an image carrying
    #4931 — a 7318 naming the phase transition, or a 7314 printed only where a registrant genuinely can
    block.
20. rbuergi

    added a commit that references this issue on Sep 19, 2026

    review(#3593): 7318's guidance is per PHASE, not one Quiescing senten…

    ...

    27b89a4
21. rbuergi

    added

    bugSomething isn't workingSomething isn't working

    area:messagingCore subsystem: message hubs - delivery, disposal, lifecycleCore subsystem: message hubs - delivery, disposal, lifecycle

    feature:hub-disposalFeature: hub disposalFeature: hub disposal

    sev:MMedium - secondary path broken, or easy workaroundMedium - secondary path broken, or easy workaround

    on Sep 20, 2026
22. rbuergi commented on Sep 21, 2026

    

    rbuergi

    on Sep 21, 2026

    Contributor

    More actions

    Cluster triage: this symptom has five distinct roots

    Reviewed together with #1540 #1547 #1548 #3243 #3593 #2833 #2480 #5064 #4859 #5136 #4847 #4883 #4888 #4961 — every one of them reports ObjectDisposedException with Autofac.Core.Lifetime.LifetimeScope.ThrowDisposedException() on top, or a teardown that did not finish. They are not one bug. The grouping, the discriminators and the evidence are committed as Doc/Architecture/DisposedScopeAndDyingHubs (added in #5159).

    Fixed — and the fix is confirmed deployed by #4847's own reading.

    Grouped in Doc/Architecture/DisposedScopeAndDyingHubs under the hub-teardown-wedge family (with #4847, #4883, #4888).

    #4847's verdict line states the #3593 fix is live ("ScheduleDrainOne asks for PreferFairness since
    #3593"), which is the drain now being queued globally rather than onto the posting thread's local
    queue — visible to every worker and to the pool's own starvation detection.

    That makes the residual readable instead of ambiguous, and it is tracked on #4847: with the drain queued
    globally, drainsAwaitingScheduler=1 + TurnsDequeued delta 0 can only mean the CLR thread pool had no
    thread to give, so M1-by-pool-starvation is now a diagnosis rather than a guess. The leading
    candidate for what saturated the pool — DrainLoop's trampoline owning a worker for an unbounded run of
    synchronous turns — is a different change from this one and belongs there.

    Leaving the closure of this one to whoever confirms no RunLevel=Started, buffer=1, exec=0 mass event
    recurs on a rolled portal; the mechanism it reported is addressed.
23. rbuergi commented on Sep 22, 2026

    

    rbuergi

    on Sep 22, 2026

    Contributor

    More actions

    The five roots — what is merged, what is in flight, what is not

    Measured against origin/main at bc7fe79551 (2026-09-22) and the fourteen issue timelines. There is no open pull request in this repository at all, so nothing is in flight to adopt; every "unfixed" row below is genuinely unowned.

    Root (Doc/Architecture/DisposedScopeAndDyingHubs)	Issues	Status	Evidence
    R1 — the host container went down ahead of the mesh	#1540, #1547, #1548 (all closed 2026-09-21)	Fixed	PR #1573, merge 5946ea4d4b (2026-08-14): MeshTeardownHostedService drains in StoppedAsync (commit a2906a04fd "fix(#1548): drain the mesh in StoppedAsync, after every other hosted service"). Pinned by MeshTeardownRunsAfterEveryOtherHostedServiceTest.
    R2 — a dying hub is handed out as the live hub	#5136 (open, sev:M)	Unfixed — no PR, open or merged, touches the lookup	Two of the three prerequisites the page names have landed: (1) the registry removal is value-matched — PR #5191, 7ce4c432cc (2026-09-22, HostedHubsCollection.Track); (2) the routing service's local route is value-matched — PR #5159, 77d8dcf5fa (2026-09-21, OrleansRoutingService.RegisterStream, Monolith twin). (3) the pod-hub cluster claim (PodHubGrain.Detach stamps a terminal tombstone regardless of who holds the address) has not landed, and GetHubWithOutcome still answers Available for a hub at RunLevel >= ShutDown. The page's "Add unreferenced" lead is settled by #5191: Add is called from MessageHubConfiguration.Build.
    R3 — a continuation resolved from a scope its own pipeline closed	#5064 (open, sev:M), #5099 (open, sev:H)	Fixed for both issues	#5064: PR #5159, 77d8dcf5fa (2026-09-21) — three delete-pipeline sites hoisted, lease-release arm reordered. #5099: PR #5163, 7a39fb2412 (2026-09-22) — the per-hub-SCOPED case, where the hoist is cosmetic; DependencyNetwork now enumerates from MeshExtensions.ReadIssuingHub. Both are compiled core code; neither has been verified on a rolled image yet, and neither thread reports a post-fix recurrence. The swept inventory (32 T1 sites, 4 fixed) is sized work with no issue attached.
    R4 — a drain budget expired over live work	#2480, #2833 (open, sev:M)	Instrumented; subject still unidentified	PR #4921, 7e78e35c9d (2026-09-19): IoPoolSiloTeardown's expiry names the pool and the leaf. RoutingQuiescenceSiloParticipant prints Stuck leg(s) at expiry (PR #4923 lineage). No occurrence since either landed has named a leaf or a leg — neither thread carries one, and the Logs action refuses from this session — so there is nothing to implement: the page and both triage notes refuse a guessed subject, and I agree.
    R5 — Orleans dropped a delivery at silo stop	#4961 (open, sev:M)	Upstream; the MeshWeaver-side question is answered in code	The one open question was whether a dropped IMessageHubGrain.DeliverMessage reaches its sender or is silently lost. It reaches the sender: RoutingGrain.DeliverToGrainRoute's Catch arm NACKs after the transient retries, and SiloUnavailableException is IsShutdownShaped → ErrorType.ShuttingDown (transient, recovery machinery stays armed) since PR #4923, fa34bd2a7b (2026-09-19). Not silently lost; no fix to write.

    Adjacent, and not one of the five roots:

    Issue	Status	Evidence
    #3593 (this issue)	Fixed, awaiting roll verification	PR #4931, b1bcbd6c86 (2026-09-19)
    #3243	Fixed, awaiting roll verification	PR #4929, e58a86e14e (2026-09-19)
    #4888	Instrumented (a direct Dispose() names its cause), caller not identified	PR #4960, 11481fed16 (2026-09-20)
    #4883	Open — the registrant is not established; it is what makes R2's window unbounded	—
    #4847	Open — the pool-telemetry step	—
    #4859	Open — Kestrel's PinnedBlockMemoryPool, not an Autofac scope; correlation step named	—

    So the one root with no fix and a fix that can be written today is R2. Next: one PR for it — the pod-hub claim made value-conditional (prerequisite 3), and the lookup no longer handing out a hub that has reached ShutDown — with a deterministic repro red on the current code. R4 stays on the instrumentation until an occurrence names its subject; R5 needs no code.
24. rbuergi commented on Sep 22, 2026

    

    rbuergi

    on Sep 22, 2026

    Contributor

    More actions

    R2 is now in flight: #5199 — the lookup retires a hub at ShutDown and mints a successor (kept in the owner's join), and the pod-hub claim's release is value-matched grain-side, which was the third prerequisite the page named. Controls on both sides are in the PR body. Per the table above, R4 stays on its instrumentation until an occurrence names its leaf/leg, and R5 needs no code; this issue itself stays open for its own post-roll verification (#4931).
25. rbuergi commented on Sep 23, 2026

    

    rbuergi

    on Sep 23, 2026

    Contributor

    More actions

    Closing: both shapes this issue carried are fixed and deployed, and neither has recurred since the fix

    This is the post-roll check the 09-22 comment left open ("this issue itself stays open for its own post-roll verification (#4931)").

    1. The titled shape is the buffered ShutdownRequest(Quiescing) never dequeued, read as RunLevel=Started, buffer=1.

    • Its mechanism, M1, was fixed by PreferFairness in ScheduleDrainOne (4c8b5815f2).
    • Its fingerprint 969aad76bce7790c has been frozen at 47 occurrences since 2026-09-07. That held across the six memex-cloud rolls of 09-19, as measured on the control instance in the 09-19 comment.

    2. The successor shape the bot folded here on 09-19 is sync/* parked at Quiescing behind Executing(ShutdownRequest, 253 s).

    • It is addressed by fix(#3593): the stall verdict below ShutDown, the unguarded phase advance, and an orphanable Executing tracker #4931 (b1bcbd6c86, merged 2026-09-19 20:03Z): the phase-guarded 7314/7318 verdicts, the Quiescing→DisposeHostedHubs Post force-faulting instead of parking, and the ExecuteOnTarget stamp cleared on every exit path.
    • git merge-base --is-ancestor b1bcbd6c86 39f49494eb is true, and 39f49494eb is what memex.meshweaver.cloud (namespace memex-cloud, where this issue's population lives) serves today (/api/version, read 2026-09-23 04:3xZ).

    3. No recurrence, and the silence was checked against a live reporter:

    • The bot's last comment on this issue is 2026-09-19 14:56Z, before fix(#3593): the stall verdict below ShutDown, the unguarded phase advance, and an orphanable Executing tracker #4931 merged.
    • The same bot posted "Still occurring" on other issues continuously up to 2026-09-23 02:30Z, so the silence is not a dead reporter.
    • REST sweep of every bot comment in this repo since 2026-09-19 20:00Z for DISPOSAL / MessageHub[731x] / RunLevel=: 0 hits.
    • REST sweep of every issue opened or updated since then for DISPOSAL DEADLOCK: only the 09-19 bodies of Hosted sync hub's disposal drain starves on an exhausted thread pool (M1-by-pool-starvation, post-#3593) #4847 and sync/ hub teardown wedges: a registered disposal cleanup blocks the ShutDown phase on the action block #4883. Neither has had a bot recurrence since.

    The residuals are not this issue and stay tracked where they are:

    • Hosted sync hub's disposal drain starves on an exhausted thread pool (M1-by-pool-starvation, post-#3593) #4847: M1-by-pool-starvation.
    • sync/ hub teardown wedges: a registered disposal cleanup blocks the ShutDown phase on the action block #4883: a registered cleanup blocking the ShutDown phase.

    What I did NOT establish:

    • I could not read Admin/_LogIncident/23170794ce38a460 directly. The systemorph MCP was down this session, and memex answered Not found, which does not separate "absent" from "not readable".
    • I did not find out when memex-cloud first rolled onto an image carrying fix(#3593): the stall verdict below ShutDown, the unguarded phase advance, and an orphanable Executing tracker #4931.

    If the successor shape recurs, the bot reopens this issue.
26. rbuergi

    closed this as completedon Sep 23, 2026