Callbacks still pending past the teardown quiescing budget (FutuRe EuropeRe, Cession, PandasExplorer) — a CreateNodeRequest the mesh hub never answers

Summary

Split out of #826. Two tests fail intermittently because a callback is still pending when teardown's quiescing budget expires — work that should have been joined, wasn't.

Test	Detail
`FutuReAnalysisTest.EuropeRe`	leaked `CreateNodeRequest` callback pending 2,738 ms past the 2 s Quiescing budget; latest run `31244442523`, 2026-08-08
`CessionLayoutAreaTest`	currently dormant (clean across 385 attempts) but its signature is identical to EuropeRe's, so it is not an independent data point — expect it to move with the same fix

Also seen in the same family and worth fixing together or immediately after:

MeshHubDisposalLeakTest.MeshHub_IsCollected_AfterMeshAndServiceProviderDisposal — 5 occurrences on 2026-08-08 alone, including on main. It was measured to be unaffected by the CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613/Unload node ALCs at the END of teardown, not before the I/O drain (#613) #967 teardown-ordering fix (identical before and after, 12 forced GCs either way), so it needs its own diagnosis. A concrete lead was recorded during that work: the test hand-rolls its teardown — hub.Dispose() → await DisposalCompleted → sp.Dispose(), setting _selfDisposed = true — so base.DisposeAsync(), which is what runs ioPools.DrainAll() and fires the terminal teardown signal, never executes. Pooled leaves can therefore still be live when it takes its GC snapshot.
SourceDocumentDataLoadingTest was seen failing with the same "left Observe subscriptions pending past the Quiescing budget" message on shard 4.

Evidence base: a sweep of 600 runs / 640 run-attempts (2026-08-04 → 2026-08-09) reading every attempt, with all 247 failing shard logs verified non-empty.

Why this is a real defect, not test noise

#826's premise — that these were environment wobble — was disproven: concurrent-run density around failing attempts is identical to the whole population, and every member of that cluster which ever recurred turned out to be a genuine defect (#834, #968, #975, #955, #932, #977, #618).

The related-but-distinct one already fixed: #613/#967 was a teardown phase inversion — node ALCs unloaded in DisposeImpl() while pooled render work was only joined after DisposalCompleted. That is fixed and pinned by NodeAlcUnloadTeardownOrderingTest. The failures here are the converse shape: not "unloaded too early" but "still running too late".

Suggested approach

DisposalCompleted ≠ all work stopped. That distinction is the through-line of the whole teardown family, so start by establishing, for each test, which callback is still in flight and why nothing joined it — rather than by widening the budget.

🚨 Forbidden as "the fix": raising the quiescing budget, adding a sleep or retry, or disabling the test. A budget that trips means something is not completing; find what.

For MeshHubDisposalLeakTest specifically, the decisive artifact would be the ClrMD root chain from one of its CI failures, which names what still holds the reference. Note that on macOS a survivor SKIPs rather than fails (ClrMD cannot attach — the #674 guard), so local behaviour cannot confirm the CI cause.

Superseded in part by #991 for one of the three members.

I filed this covering FutuReAnalysisTest.EuropeRe, the dormant CessionLayoutAreaTest, and MeshHubDisposalLeakTest together, on the grounds that they shared a signature (a callback still pending past the teardown quiescing budget). MeshHubDisposalLeakTest now has an actual root cause and its own issue (#991) — ClrMD named the chain: a pending single-shot Observable.Timer in ActivityControlPlaneExtensions (reached via NodeTypeCompilationHelpers) sits on the process-wide TimerQueue and pins the AccessAssignment MessageHub past Mesh.Dispose().

So track that one at #991, which has the evidence. This issue keeps the other two.

That also corrects the lead I recorded here. I noted that the test hand-rolls its teardown — hub.Dispose() → await DisposalCompleted → sp.Dispose() with _selfDisposed = true, so base.DisposeAsync() never runs DrainAll(). That observation is true, but it is not the cause: a TimerQueue entry is not something DrainAll() would join, so the hand-rolled teardown is a red herring for this member. Worth stating plainly so nobody spends time on it.

Whether the remaining two share the timer mechanism or are genuinely a quiescing-budget problem is still open — they should be checked against #991's finding before assuming they are the same defect.

Re-measured on current `main`: still live after #996, and it is not #613

I was asked to establish whether this and #613 share a mechanism. They do not — the proof is on #613; the short version is that #613's crash resolves to a null MethodTable inside WKS::gc_heap::background_sweep() (twice, at instruction granularity), which is runtime-internal GC bookkeeping and has no path to a pending hub callback. Recording here what that re-measurement establishes about this issue.

#996 did not fix it — and the family is wider than FutuRe

The obvious hypothesis after #991/#996 was that this is the same Rx-timer-rooting class. It is not:

Run 31306381242, main @ 048dd80da — which is the #996 merge commit — failed with this exact signature, in a new member:

System.InvalidOperationException : PandasExplorerLayoutAreaTest left Observe subscriptions
pending past the Quiescing budget. …
Hub mesh/8bMfyuzUL0K4GIYg3sUawA: 1 pending callback(s) after 2.00s:
  URvJw1O9ikaADdmXjqkpCA=CreateNodeRequest@mesh/8bMfyuzUL0K4GIYg3sUawA(6153ms)

So the roster is no longer FutuRe-shaped: FutuReAnalysisTest.EuropeRe, CessionLayoutAreaTest, SourceDocumentDataLoadingTest, and now PandasExplorerLayoutAreaTest in Hosting.Monolith.Test.

The two captured instances share a precise signature

AppendQuiescingTimeoutSummary names the hub that registered the callback, so in both cases the pending Observe belongs to the mesh hub itself, on a request addressed to itself:

	EuropeRe (`31244442523`, 08-08)	PandasExplorer (`31306381242`, 08-09)
pending on	`mesh/qDTjb0Yj5Ey…`	`mesh/8bMfyuzUL0K4…`
request	`CreateNodeRequest@mesh/<self>`	`CreateNodeRequest@mesh/<self>`
age at the 2 s budget	2 738 ms	6 153 ms

The Pandas capture also has the full tree snapshot, and it is worth reading closely:

Hub mesh/8bMf… RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,exec=0,openGates=0)
HostedHubs (1):
  Hub Doc/DataMesh/PythonPandasNode/PandasExplorer RunLevel=Quiescing Disposal=Pending
    PendingCallbacks=1[…=CreateOrUpdateNodeRequest@mesh/8bMf…(969ms)]

Two facts that constrain the cause:

The mesh hub's callback predates disposal. Dispose() was invoked at 09:49:51.134; the snapshot at 09:49:54.135 shows the request already 6 153 ms old, i.e. posted ~3.1 s before teardown began, and unanswered that whole time while the mesh was fully alive. Whatever this is, teardown only reveals it — the budget expiring is the detector, not the defect.
The child is then structurally unable to finish. By the time PandasExplorer reaches Quiescing it has its own CreateOrUpdateNodeRequest outstanding to the mesh hub, which by then is at DisposeHostedHubs — past the run level at which it processes work. Every queue is empty (buffer=0, deferred=0, exec=0), so nothing is wedged on an action block; the reply simply has no one left to produce it.

That second point is a parent/child ordering hazard worth naming on its own: a child that still needs the parent is quiesced after the parent has stopped answering. Whether it is the cause here or a consequence of (1) is exactly what is still open.

A lead I would not push too hard: in the EuropeRe capture the framework's own ROUTER_TRAFFIC guard fires for this traffic — "CreateNodeRequest has the mesh hub as target (sender: FutuRe/LineOfBusiness) … the mesh hub is the ROUTER and must not execute work" — i.e. the stuck callbacks sit on traffic the framework already declares invalid. The Pandas capture shows no such lines, so I am recording it as a correlation on one member, not a shared cause.

Corrections to this issue's own framing

The MeshHubDisposalLeakTest member is gone from here (→ A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991, fixed by A pending Rx timer kept disposed hubs alive — in four watchers, three of them on the live thread path #996), and with it the "hand-rolled teardown skips DrainAll()" lead, which I had already withdrawn as non-causal. Both of the remaining members have now been checked against A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991's finding, as that comment asked: neither is the timer mechanism — the pending item is a hub Observe callback awaiting a reply, not a TimerQueue entry, and one reproduced on the A pending Rx timer kept disposed hubs alive — in four watchers, three of them on the live thread path #996 merge commit itself.
"the converse of CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613's phase inversion" in the title should go. CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613's exit=134 phase inversion is real and fixed, but this issue's members are not its converse — they are a message-lifecycle defect with no ALC, GC or native surface at all. Keeping the two framed as a pair is what made "are these one defect?" a live question in the first place.

Still open, and what I did not do

I did not root-cause why that CreateNodeRequest goes unanswered for 3–6 s on a live mesh — that is the actual question, and it deserves its own pass rather than a guess appended to a triage. The next step I would take is the cheap one: make the failure name the handler side as well as the caller side, so a capture says whether the request was received and never answered, answered to a dead callback, or never routed. Right now every capture we have is caller-side only, which is why two independent occurrences still leave the mechanism unnamed.

🚨 Unchanged: raising the 2 s budget is not a fix. The request is unanswered for three times the budget while the mesh is still fully alive; a bigger budget just moves the detector.

Retitled both issues to match the evidence, because the old titles are what kept sending investigations the wrong way — and I want the correction to be visible without reading thirteen comments first.

CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613 was "Teardown SIGSEGV in MeshWeaver.FutuRe.Test (exit=139, no ClrMD) — collectible-ALC unload suspect". Three dumps now say it is neither a teardown crash (one of the three lands mid-Dispose(), two land inside a fresh test) nor an ALC unload (a freed LoaderAllocator is a non-null unmapped pointer; this faults on RAX = 0 reading a MethodTable at address zero). It is WKS::gc_heap::background_sweep(), RVA 0x5cb1e1 on runtime 10.0.10, every time.
Callbacks still pending past the teardown quiescing budget (FutuRe EuropeRe, Cession, PandasExplorer) — a CreateNodeRequest the mesh hub never answers #981 was "… (FutuRe EuropeRe, Cession, MeshHubDisposalLeakTest) — the converse of CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613's phase inversion". MeshHubDisposalLeakTest moved to A pending Rx timer in ActivityControlPlaneExtensions pins the AccessAssignment hub past Mesh.Dispose() #991 and is fixed; PandasExplorerLayoutAreaTest joined; and the "converse of CoreCLR heap corruption: an object's MethodTable word is zeroed; the GC faults walking it (upstream — not fixable here) #613" framing is what made "are these one defect?" a live question — they share no mechanism.

Nothing about either issue's substance changed, only the label on the box.

A third capture — and it narrows the root cause to the parent's stuck request

This reproduced on my own PR #1004 an hour after the last comment. That PR's diff is one markdown file — zero C#, nothing that could create a callback — which is another confirmation that this is pre-existing and load-dependent rather than anything a branch introduces.

Run 31312455102, shard 4 — a fifth member, in a third project: ContentCollectionReferenceTest (MeshWeaver.Content.Test).

ContentCollectionReferenceTest left Observe subscriptions pending past the Quiescing budget.
Hub mesh/hI5K7RnK_EW7Fb-CZyeznA: 1 pending callback(s) after 2.00s:
  i5_bb_j0H0uiTbW5oOPcXg=CreateNodeRequest@mesh/hI5K7RnK_EW7Fb-CZyeznA(3168ms)

[12:19:28.712] Mesh.Dispose() invoking
[12:19:31.712] still waiting after 3000ms — snapshot:
  Hub mesh/hI5K7RnK… RunLevel=DisposeHostedHubs Disposal=Pending Queue(buffer=0,deferred=0,exec=0,openGates=0)
  HostedHubs (1):
    Hub Cornerstone/Insured RunLevel=Quiescing Disposal=Pending Queue(buffer=0,deferred=0,exec=0,openGates=0)
      PendingCallbacks=1[…=CreateOrUpdateNodeRequest@mesh/hI5K7RnK…(992ms)]
      HostedHubs (34): …all sync/… RunLevel=Started Disposal=<not started>

Structurally identical to the PandasExplorerLayoutAreaTest capture: mesh hub at DisposeHostedHubs holding a CreateNodeRequest to itself; exactly one hosted node hub at Quiescing holding a CreateOrUpdateNodeRequest to the mesh; that node hub carrying 34 sync/ children, all still Started; every queue empty. Different partition (Cornerstone/Insured vs Doc/DataMesh/PythonPandasNode/PandasExplorer), different project, same picture.

The child's stuck request is a consequence, not the defect — the arithmetic says so

Lining the two captures up against their own Dispose() timestamps:

	Dispose invoked	snapshot	child callback age	⇒ child posted at
ContentCollectionReference	`12:19:28.712`	`12:19:31.712`	992 ms	T + 2 009 ms
PandasExplorer	`09:49:51.134`	`09:49:54.135`	969 ms	T + 2 032 ms

Both land at T + ~2 000 ms, which is exactly QuiesceTimeout. That is not a coincidence, it is the state machine:

Dispose() → mesh hub enters Quiescing with its CreateNodeRequest already outstanding.
At T+2 s the quiesce deadline fires (Observable.Timer(QuiesceTimeout) racing the drain via Amb), QuiescingTimedOut = true, and the hub advances to DisposeHostedHubs → hostedHubs.Dispose().
Only now does the child node hub start quiescing — and in doing so it posts CreateOrUpdateNodeRequest to the mesh hub, which is by then past Started and will never answer it.

So the child's pending callback is downstream of the parent's timeout, generated by it. Chasing the parent/child ordering hazard I flagged last time would be treating the symptom.

What that leaves as the actual question

Why is the mesh hub's own CreateNodeRequest unanswered before disposal even starts? In all three captures it was posted before Dispose() and was already stuck:

capture	age at report	⇒ posted before `Dispose()` by
EuropeRe	2 738 ms	~740 ms
ContentCollectionReference	3 168 ms	~1 170 ms
PandasExplorer	6 153 ms	~3 150 ms

A live, idle mesh (buffer=0, deferred=0, exec=0, openGates=0, deliveryActionCompleted=True) is sitting on an unanswered CreateNodeRequest for 0.7–3.2 seconds before anything is torn down. Nothing is wedged on an action block — the reply is simply not coming.

That is the whole defect, and it is worth restating that the quiescing budget is only the detector. Raising it would not have made any of these five tests pass; it would have made the mesh wait longer before recording the same unanswered request.

The one instrument that would end this

Every capture we have is caller-side only — it names the request, its target and its age, and says nothing about whether the mesh hub ever received it, dispatched it, or produced a response that went nowhere. Three independent reproductions have now produced the same caller-side line and no more. Pairing the pending-callback snapshot with the responder's view (received / handled / responded, or never routed) is what turns the fourth occurrence into a root cause instead of a fourth identical screenshot.

Narrowing, from the captures themselves — two hypotheses dead, one mechanism confirmed in code

Recording these so the next person doesn't re-walk them. All three are readable off evidence already in this issue; none needed a new run.

Ruled out — the CreateNodeRequest init gate. MeshDataSource.cs:182 puts an unbounded initialization gate on exactly this message type (.WithInitializationGate(MeshNodeInitGateName, d => d.Message is CreateNodeRequest)), and MeshNodeTypeSource's docstring says out loud that activation has no wall-clock bound on purpose — "a cold-partition read under a boot storm legitimately takes tens of seconds", with a CreateNodeRequest "still in flight behind the init gate" named as an expected state. That is a near-perfect fit for "a create sits unanswered for seconds", and it is not what happened: GetQueueSnapshot() reports openGates = gates.Count and deferred = deferredQueue.Count, and every hub in both snapshots shows deferred=0, openGates=0. Nothing was held behind a gate.

Ruled out — Observable.FromAsync in the create path. HandleCreateNodeRequest's own summary still claims "All async work is wrapped in Observable.FromAsync", which would be both a rule violation and a plausible stall (unbounded, prologue on the subscribing thread). The comment is stale — the current code notes the storage call is already IObservable "so we don't need to wrap it in Observable.FromAsync". Worth fixing that docstring on the way past, but it is not the defect.

Confirmed in code — the child's stuck request is dropped, not merely unanswered. My timing argument (the child posts at T+2 s, i.e. the moment the parent's quiesce budget expires) has a direct mechanical counterpart in MessageService.ScheduleNotify:

"During shutdown, only allow ShutdownRequest and DisposeRequest through. All other messages (including DeliveryFailure) are dropped to prevent endless cascades."

So the child's CreateOrUpdateNodeRequest, posted to a mesh hub already in shutdown, is discarded by design — its callback can never complete, and no DeliveryFailure comes back either. That is a consequence of the parent timing out, not an independent leak, and it will disappear when the parent's request stops hanging.

What that leaves, stated as sharply as the evidence allows. On an otherwise idle mesh (buffer=0, deferred=0, openGates=0, deliveryActionCompleted=True, nothing executing), the CreateNodeRequest was dequeued and handled — the handler returns request.Processed() immediately by design and posts the response from inside a composed observable. So the response is owed by that composed chain, and for 0.7–3.2 s it does not arrive. The remaining question is therefore narrow: does that chain complete slowly, or does it terminate without ever posting a response? Those two have different fixes, and the caller-side snapshot cannot tell them apart — which is the same reason I asked for responder-side instrumentation above.

The failure now reports the HANDLER side — PR #1011

Taking the next step named at the end of the previous comment ("make the failure name the handler side as well as the caller side"). This is instrumentation only — no budget change, no retry, no response posted anywhere. The issue stays open.

What was missing

Every capture so far answers "what did this hub post and never hear back about". None answers "and what happened to that delivery". RequestFateLedger records, per awaited request id, the stages the delivery actually reached, and folds the trail into every place a pending callback is already reported — the teardown [QUIESCE-TIMEOUT], the live-mesh [STALE-CALLBACK] scanner, and GetDisposalDiagnostics.

A capture now reads like this (from the regression test that pins it):

Pending: WqgJ…=MisroutedReplyRequest@host/1(303ms)
  handler-side fate (what happened to the delivery):
    WqgJ…=MisroutedReplyRequest:
      POSTED target=host/1@client/ccfc…(+0ms) → RECEIVED runLevel=Started@client/ccfc…(+0ms)
      → ROUTED onTarget=False state=Forwarded@mesh/1(+1ms) → RECEIVED runLevel=Started@host/1(+1ms)
      → ROUTED onTarget=True state=Submitted@host/1(+1ms) → HANDLER_ENTER@host/1(+1ms)
      → RESPONSE_POSTED type=SwallowedResponse target=mesh/1@host/1(+1ms)
      → HANDLER_EXIT state=Processed@host/1(+2ms)
      → RESPONSE_ARRIVED_NO_SUBJECT type=SwallowedResponse@mesh/1(+2ms)
      ⇒ a reply WAS posted for this correlation and the callback is STILL pending — the reply was
        lost between the responder and the requester, so chase the response delivery, not the handler.

Each trail ends in a verdict naming which of the mutually exclusive shapes it is: never delivered / delivered-but-parked (naming the gate, breaker or shutdown check) / no handler matched / handler ran and never replied / chain completed without producing a reply / reply posted and lost.

The binary question it was built to settle

Does the chain that owes the reply complete slowly, or terminate without responding? Three parts close it:

HANDLER_COMPLETED_WITHOUT_DELIVERY — a rule chain that completes without emitting is silent in every other diagnostic and reads exactly like one still running.
hub.NoteRequestStage(requestId, stage) — the seam a handler uses to report on work it detached from the rule chain. This matters because the canonical mesh handlers return request.Processed() immediately and owe their reply from a composed observable, so the pipeline's own HANDLER_EXIT proves nothing. HandleCreateNodeRequest — the handler named in every capture — now records CREATE_CHAIN_SUBSCRIBED / _EMITTED / _ERROR / _COMPLETED_EMPTY plus the post-creation arms.
The verdict, so a reader gets the answer rather than raw stages to interpret.

One code-reading observation — a candidate, NOT a proven cause

While adding the terminal arms I found that HandleCreateNodeRequest's outer .Subscribe(onNext, onError) has no onCompleted arm, and both save paths filter with .Where(n => n is not null):

var saveObs = persistence != null
    ? persistence.WriteAndPublishCreated(enriched, …).Where(n => n is not null).Select(n => n!)
    : Observable.Return(enriched);

If that write ever emits null, the chain completes empty, no response is posted, and the caller's callback stays pending forever — which matches this issue's signature (CreateNodeRequest@mesh/<self>, unanswered, every queue empty, no handler wedged). Several other branches also return Observable.Empty<…>(); most post a Fail first, so they are answered, but the outer subscription has no backstop if any of them does not.

I am deliberately not calling this the root cause and not fixing it. It is a code reading, not a capture. The new onCompleted arm records CREATE_CHAIN_COMPLETED_EMPTY and deliberately does not post: which response is correct depends on why the chain went empty, and that is unknown until a real capture names the branch. Posting one now would be a guessed fix.

What it rules in and out

Rules IN, as a testable hypothesis: an empty-completing create chain. The next capture will say CREATE_CHAIN_COMPLETED_EMPTY outright if so.
Rules OUT by construction, once a capture lands: "never delivered", "no handler matched", "parked behind a gate", and "reply posted but lost" — each has its own stage and its own verdict, so none of them can be confused with the others any more.
Still cannot see: a reply sent under the wrong correlation reads as HANDLER_EXIT with no RESPONSE_POSTED, i.e. the same as a handler that replied to nothing. The tell is a spurious RESPONSE_POSTED on a different request's trail.

Where the evidence lands

QuiescingTimeoutDetail → GetQuiescingTimeoutSummary() → MonolithMeshTestBase's DISPOSE_QUIESCE_LEAK phase trace → /tmp/meshweaver-test-trace.log, collected by dotnet-test.yml with if: always(). All the reported members derive from MonolithMeshTestBase.

Two traps checked rather than assumed:

Not suppressed by Bound the trace log's fault RATE, not its lifetime total — a spent budget silenced the wedge #994's rate budget — TestPhaseTrace writes via TestTraceLog.Append, not AppendFault, so FaultRecordBudget does not apply and no [FAULT-BUDGET] marker can swallow these records.
Nothing to misconfigure (Three read-after-write flakes, three unrelated defects (#978) #997's duplicate-appsettings.json failure, where the diagnostic emitted locally and never on CI) — TestTraceLog writes to a fixed path and reads no configuration.

The trail is also rendered by [STALE-CALLBACK], which fires while the mesh is live. That matters for this issue's first hard constraint — the callback predates Dispose() by ~3.1 s — so a repro run that dials MESHWEAVER_STALE_CALLBACK_MS down gets the handler side before teardown is involved at all.

Also corrected

HandleCreateNodeRequest's doc-comment claimed its async work was wrapped in Observable.FromAsync. It is not, and that pattern is now forbidden outside IoPool — a stale doc advertising a banned pattern is one copy away from becoming true.

Stop the zero-pending quiesce report from reading like a leak

Two things the local trace turned up while validating the channel

Recording both, because a grep DISPOSE_QUIESCE_LEAK over a CI artifact hits them and neither is this issue.

1. A locally reproducible member of this failure class — and it is not a create. SpaceMenuAndAccessTest produces, repeatedly:

DISPOSE_QUIESCE_LEAK elapsed=5509ms Hub mesh/Neb3wo0mV0mY-CMgO1EXLg: 2 pending callback(s) after 0.50s:
  eGm-71eGSUC-vYbqjuHs-Q=DeleteNodeRequest@TestSpace_ccdf91d116/Overview(10420ms),
  ca-l9pfV3kOu6VpqGYwC-Q=DeleteNodeRequest@TestSpace_ccdf91d116/_Access/Roland_Access(10420ms)

Same shape as this issue — a node-lifecycle request unanswered for 10.4 s on a live mesh, teardown merely detecting it — but DeleteNodeRequest to a per-node hub rather than CreateNodeRequest to the router. It reproduces locally, unlike the create case, so it is the cheaper place to get the first handler-side trail once #1011 lands. (It looks like the already-known space-delete child-hub wedge; whether that and this are one defect is exactly what a trail would settle.)

2. A false positive that looks identical at a glance. 26 of the 30 leak records in the same trace say:

DISPOSE_QUIESCE_LEAK elapsed=4008ms Hub mesh/iHkClK3K6kqtgx2f5k9-bg: 0 pending callback(s) after 2.00s: <none>

The poll reported not-drained; by the time OnQuiesceComplete rendered the verdict, responseSubjects was empty. Every callback resolved inside the hand-off between the last poll tick and the verdict — nothing leaked, the detector raced its own budget. It still sets QuiescingTimedOut and still fails the class, and <none> reads like a diagnostic that failed to collect its evidence, so it sends an investigation hunting a leak that was never there. Three classes produce it locally: TodoDataChangeWorkflowTest (12), TodoGraphIntegrationTest (9), TodoViewsTest (5).

#1011 makes that case say what actually happened instead of 0 pending callback(s): <none>. Behaviour is unchanged — same flag, same failure — only the sentence, so the two shapes can be told apart by grep. Whether the detector should fail a class when nothing is outstanding is a separate question and deserves its own issue; I have not changed it here.

Seed the trail at registration; render it on one greppable line

Close the last ambiguity: an Observe that registers but never posts

The shutdown reject's classification had nothing holding it in place

Not the same defect as #1029 — and here is the proof, plus what it rules out here

I was asked to establish whether this and #1029 are one defect. They are two. #1029 is now
root-caused and fixed (PR below); recording the disproof here because the hypothesis it kills —
"something between a DeliveryFailure/response and the awaiting Observe drops or mis-correlates
it" — is the one this issue would most naturally inherit next.

What #1029 turned out to be

A read that raced a just-deleted node's hub teardown. Reproduced deterministically
(PostDeleteReadVerdictTest); the MESHWEAVER_MSG_TRACE capture names every step:

17:51:12.884  TestData/…  RawJson id=Rqn…  ScheduleNotify ENTER runLevel=DisposeHostedHubs
17:51:12.884  TestData/…  RawJson id=Rqn…  DROPPED_SHUTTING_DOWN
17:51:12.891  mesh/…      SubscribeRequest id=Rqn…  routed state=Forwarded isOnTarget=False
17:51:12.892  mesh/…      DeliveryFailure  id=Cu1…  NotifyAsync ENTER state=Submitted
17:51:12.899  cache/…     DeliveryFailure  id=Cu1…  routed isOnTarget=True

The reply was posted, was routed, and did arrive at the requester. It was then swallowed
deliberately by JsonSynchronizationStream's ErrorType.ShuttingDown ride-out, which keeps the
stream alive for a change-feed resubscribe latch that a deleted node will never trigger.

So the shared-mechanism hypothesis is falsified for #1029: nothing was lost between responder
and requester, and nothing was mis-correlated. The failure was a consumer acting correctly on a
mis-classified answer.

That leaves nothing shared with this issue's signature, which has none of those parts: a
CreateNodeRequest on-target to the mesh hub itself, on a mesh that is fully alive
(buffer=0, deferred=0, exec=0, openGates=0), with no sync stream, no routing hop, no drop and no
NACK anywhere in the picture. Different layer, different mechanism.

What that means for the `RequestFateLedger` verdicts to expect here

Two of the ledger's shapes can now be set aside for this issue on evidence rather than assumption:

RESPONSE_POSTED with the callback still pending ("the reply was lost between responder and
requester") — that is the shape An authoritative routing NotFound is produced but never becomes a verdict — the reader waits out its full budget #1029 looked like from the outside and provably was not; there is
no analogous hop in this issue's captures at all (sender and target are the same hub).
RESPONSE_ARRIVED_NO_SUBJECT — likewise needs a delivery crossing hubs to land on the wrong
one. On-target self-addressed traffic has nowhere else to land.

The verdict this issue should still be read as expecting is CREATE_CHAIN_COMPLETED_EMPTY.

One thing I can add to that candidate: it is REACHABLE, not just a code reading

The previous comment recorded the empty-completion candidate honestly as "a code reading, not a
capture". It is worth upgrading that to reachable, because the null it depends on is a
documented sentinel rather than a hypothetical:

IStorageAdapter.Write returns IObservable<MeshNode?> and null is the try-then-claim
sentinel — "this adapter does not own this path". StorageAdapterChangeFeedExtensions says so
in as many words ("Skips the publish if the adapter emits null — that's the try-then-claim sentinel
meaning 'this adapter does not own this path,' not 'the write succeeded'"), and
PathFilteringStorageAdapter.Write returns exactly that for a non-matching path. Both save paths in
HandleCreateNodeRequest then apply .Where(n => n is not null), so that sentinel terminates the
chain with no emission and no response posted — the requester's callback stays pending forever.

Note the asymmetry that hides it: the composite PersistenceService.Write converts the same
condition into a throw ("Could not save '…': no writable storage provider accepted the node"),
which the handler's onError arm answers correctly. So whether a create hangs or fails cleanly
depends on which adapter is resolved — which is exactly the kind of difference a load-dependent CI
flake would expose and a local run would not.

I have not proven this is what the captures hit, and I did not "fix" it: posting a guessed
response from the onCompleted arm is what the previous comment deliberately declined to do, and
that judgement still looks right. What a capture needs to say is which branch went empty.

What I could not reproduce

PandasExplorerLayoutAreaTest run locally with MESHWEAVER_STALE_CALLBACK_MS=700 (so the live-mesh
scanner would report a pending callback well before teardown, per the instrumentation note) passed
in 1 s with no [STALE-CALLBACK] and no pending CreateNodeRequest. One run, not repeated — a
re-run to "see if it flakes" is exactly what hides the race. So this stays load-dependent and
CI-only for now, and the next CI occurrence with #1011's ledger attached is still the thing that
names the branch.

The empty-completion lead, stated as a defect in its own right — and exactly what capture confirms it

Follow-up to my previous comment, pinning down the part that is worth keeping regardless of what
the next CI occurrence says.

It is a defect whether or not it is THIS issue's captured failure

IStorageAdapter.Write returns IObservable<MeshNode?> where null is a documented
try-then-claim sentinel — "this adapter does not own this path". StorageAdapterChangeFeedExtensions
says so in as many words:

Skips the publish if the adapter emits null — that's the try-then-claim sentinel meaning "this
adapter does not own this path," not "the write succeeded."

PathFilteringStorageAdapter.Write returns exactly that for a non-matching path. Both save paths in
HandleCreateNodeRequest then apply .Where(n => n is not null), so that sentinel terminates the
chain with no emission and no response posted, and the requester's callback stays pending
forever.

The asymmetry is the tell. The composite PersistenceService.Write turns the same condition
into a throw — "Could not save '{path}': no writable storage provider accepted the node." —
which HandleCreateNodeRequest's onError arm answers correctly with a Fail. So one and the same
condition either fails cleanly or hangs forever, decided by which adapter instance the
handler resolved. That is a defect on its own terms: whether a caller gets an answer must not depend
on the storage wiring underneath it. It also explains the load-dependence — the hanging shape needs a
non-composite adapter in the resolved position, which is a configuration a local single-partition run
does not necessarily produce.

What capture confirms it — stated concretely so the next occurrence is not wasted

#1011's ledger already emits everything needed; the confirmation is a conjunction, not a single
line. In the [QUIESCE-TIMEOUT] / [STALE-CALLBACK] block for the pending
CreateNodeRequest@mesh/<self>, the trail must show all of:

CREATE_CHAIN_SUBSCRIBED path= — the handler ran and detached its chain;
no CREATE_CHAIN_EMITTED — the save never produced a node;
no CREATE_CHAIN_ERROR — it did not fault (which would have been answered);
CREATE_CHAIN_COMPLETED_EMPTY path= — the terminal arm Report the HANDLER side when a callback outlives the quiescing budget #1011 added;
no RESPONSE_POSTED anywhere on that trail — nothing answered the correlation.

⇒ verdict text: "the chain COMPLETED WITHOUT PRODUCING A REPLY … look for a filtering operator (a
Where that dropped the only element, an Observable.Empty branch) upstream of the post."

That combination is sufficient: (2)+(3)+(4) together can only be produced by an upstream that
completed empty, and the two save paths are the only empty-capable branches that do not post a
Fail first (the already-exists / validation-failed / invalid-NodeType branches all post before
returning Observable.Empty, so any of those would show a RESPONSE_POSTED).

What it still will not say is WHICH .Where dropped it, and that is the one gap worth closing
cheaply before the next occurrence: a stage recorded at the null-emission itself — e.g.
CREATE_SAVE_DECLINED adapter=<type> path= noted from a .Do(n => { if (n is null) … }) ahead of
each .Where(n => n is not null) — would name the adapter that declined and turn the fourth capture
into a fix instead of a fourth confirmation. That is instrumentation, not a behaviour change, and it
is strictly narrower than guessing a response.

Still not fixing it here

Posting a response from the onCompleted arm remains a guess about which response is correct, which
is what the previous comment declined to do and I agree with. The honest options once a capture
lands are (a) make the null-emission a Fail carrying the declining adapter — aligning the
non-composite path with the throw PersistenceService already produces — or (b) remove the
asymmetry at the source so a decline is never silent. Both are one-line-ish and both need the
capture to say which adapter is in the resolved position.

Unchanged

The An authoritative routing NotFound is produced but never becomes a verdict — the reader waits out its full budget #1029 shared-mechanism hypothesis stays falsified (reply posted, routed, delivered — the trace
is in my previous comment); An authoritative routing NotFound is produced but never becomes a verdict — the reader waits out its full budget #1029 is fixed in PR A deleted address must answer NotFound, not the transient ShuttingDown #1038.
PandasExplorerLayoutAreaTest locally with MESHWEAVER_STALE_CALLBACK_MS=700 passed in 1 s with
no [STALE-CALLBACK]. One run, not repeated.

The empty-completion lead is now fixed as a defect in its own right — and it is provably NOT this issue's captures

PR #1050 (open) does the two things the previous comment listed as the honest options, and adds the instrumentation that comment asked for. It is Refs #981, deliberately not Fixes — the reasoning below is the part worth keeping here.

What shipped

The decline is no longer silent. RequireClaimedWrite replaces both save paths' .Where(n => n is not null). The try-then-claim null sentinel now faults with the same InvalidOperationException message family the composite PersistenceService.Write already throws, so both paths produce an identically shaped CreateNodeResponse.Fail(…, ValidationFailed) from the one onError arm. One condition, one answer, regardless of storage wiring.
CREATE_SAVE_DECLINED adapter=<type> path= is recorded at the null emission — the "which .Where dropped it" gap named at the end of the previous comment, closed exactly as proposed and no wider.
The terminal onCompleted arm now answers. Gated on emitted (the chain produced a node, so the post-creation Subscribe owes the reply) and responded (a branch already posted its own, more specific rejection). Every terminal post goes through one local Respond(…), which is what makes the gate exact rather than approximate.

Pinned by CreateNodeAlwaysAnswersTest, negative control verified: with only the handler reverted, the two hang tests fail after the full 15 s with "The observable emitted nothing at all" — a hang, which is today's behaviour — while the already-exists and success guards pass in both states.

Why this is not this issue's captured failure

All five reported members derive from MonolithMeshTestBase → AddInMemoryPersistence(), whose resolved IStorageAdapter is

SubtreeDeletionGuard( MonotonicWriteGuard( VersionWriting( InMemoryStorageAdapter ) ) )

InMemoryStorageAdapter.Write never emits null — it returns the node, or the version winner on a refused backward write — and all three decorators only forward a null they were handed. So the sentinel branch is unreachable in the exact wiring every capture came from. The decline fix is a real adjacent defect; it is not the one the captures hit.

What the fix does do for this issue

If the create chain is what completes empty, the backstop now answers, the pending callback resolves, and the [QUIESCE-TIMEOUT] symptom disappears — leaving an ERROR log naming the path plus a CREATE_CHAIN_COMPLETED_EMPTY ledger stage instead of a sixth identical caller-side screenshot. If the chain is instead still running (a leaf that never completes), the backstop never fires and this issue is untouched. Nothing in the captures distinguishes those two yet, which is precisely why this is Refs.

One new lead, from reading the Monolith wiring

With the save ruled out for that wiring, the strongest remaining empty-completion candidate inside the create chain is EnsurePartitionBootstrap →

hub.CheckPermission(partition, effectiveUser, Permission.Create)
    .TakeDecisionOutsideGate()          // == decision.Take(1).SelectMany(HopOffGate)
    .Timeout(TimeSpan.FromSeconds(15))
    .Catch(...)                          // fault → false → heal skipped, chain continues
    .SelectMany(authorized => ...)

If the permission fold completes without emitting, Take(1) yields nothing → the SelectMany chain goes empty → EnsurePartitionBootstrap emits nothing → the whole create completes empty → no response. Two details make this fit rather than merely being possible:

.Timeout(15s) does not catch it. Timeout faults on a stall, not on an empty completion, so an immediate empty completion passes straight through — and the Catch below it only ever sees the stall case.
The arithmetic fits. The captures show the CreateNodeRequest unanswered for 0.7–3.2 s before Dispose(), i.e. well under the 15 s bound. A stall would have tripped the timeout and been answered false; an immediate empty completion would look exactly like what was captured.

hub.CheckPermission rides GetEffectivePermissions, a seed.Concat(enriched) fold over cached Replay(1) synced queries — an upstream that completes is what makes the fold complete. I have not proven it happens; this is a code reading, same standing as the save lead was two comments ago. The difference is that it is no longer silent: the next occurrence will carry CREATE_CHAIN_COMPLETED_EMPTY path= and an ERROR line naming the path, and the caller will have been answered rather than left pending.

Unchanged

The quiescing budget is still only the detector; nothing here raises it.
The child's CreateOrUpdateNodeRequest pending at T+2 s remains a consequence of the parent's timeout (ScheduleNotify drops it during shutdown), not an independent leak — untouched, and expected to disappear with whatever finally explains the parent.

Split "the permission fold is slow" from "it terminated" in the creat…

The `TakeDecisionOutsideGate` lead: RULED OUT — and ruling it out reframes the issue

I named this candidate two comments ago and said it was a code reading, not a capture. Having now
followed it to the bottom, it cannot happen through the shipped evaluator — and the reason it
cannot is the same reason the other branch of the binary question is the live one.

The structural worry was real

EnsurePartitionBootstrap's authorization probe is

hub.CheckPermission(partition, effectiveUser, Permission.Create)
    .TakeDecisionOutsideGate()        // == decision.Take(1).SelectMany(HopOffGate)
    .Timeout(TimeSpan.FromSeconds(15))
    .Catch(... => Observable.Return(false))
    .SelectMany(authorized => ...)

.Timeout faults on SILENCE, not on a clean finish. A fold that completes without emitting
passes straight through the Timeout, never triggers the Catch behind it, and vanishes through
the SelectMany — so EnsurePartitionBootstrap would emit nothing and the create's chain would
terminate unanswered. That is the identical mechanism to the .Where(n => n is not null) in
#1050, which is exactly why it deserved chasing.

Why it cannot fire

The fold bottoms out in SyncedQueryMeshNodes, whose stream is

var allChanges = upstream.Merge(externalChanges).Merge(feedRemovals);

externalChanges is a Subject<QueryResultChange<MeshNode>> on which OnCompleted is never
called — there is no such call site in the file — and Merge completes only when every source
completes. So allChanges can never complete, therefore the synced query can never complete,
therefore cache.GetQuery(...) can never complete-without-emitting, therefore the four
CombineLatest sources in PermissionEvaluator can't either, therefore seed.Concat(enriched)
cannot complete empty.

The fold can only stall — and a stall is precisely what the .Timeout(15 s) does catch
(→ Catch → false → heal skipped → the create is answered).

The empty case survives in exactly one place: EffectivePermissionsDelegate is a DI extension
point, and an evaluator returning Observable.Empty<Permission>() is a legal implementation. That
is a guard case, not a hot path.

What that reframes about this issue

The permission fold is bounded at 15 s on purpose — a cold-start synced query on a
freshly-provisioned partition legitimately takes seconds. The teardown quiescing budget that
detects the pending callback is 2 s.

So a create caught mid-probe is reported as a leaked callback long before its own bound would have
fired. That fits every capture's arithmetic directly: 0.7 s, 1.2 s and 3.2 s are all comfortably
under 15 s. On this evidence the captures are the "the chain completes slowly" branch of the
binary question this issue posed — not the "terminates without responding" branch I had been
weighting.

If that is right, the honest consequence is uncomfortable and worth stating plainly: there may be
no leak here at all — a create legitimately waiting on a cold-start authorization probe, detected
by a budget an order of magnitude shorter than the wait it is observing. I am not asserting that;
raising the budget remains the wrong move and I have not done it. What I am asserting is that the
next capture can now say which, instead of being ambiguous.

The instrument that settles it — PR #1050 (branch updated)

BOOTSTRAP_PERM_AWAIT partition=… user=… before the fold, then exactly one of:

Stage	Meaning
`BOOTSTRAP_PERM_VERDICT authorized=…`	the fold answered
`BOOTSTRAP_PERM_FAULTED <Type>`	it stalled past 15 s (or threw) and was classified
`BOOTSTRAP_PERM_COMPLETED_EMPTY …`	it terminated with no verdict — the case `Timeout` cannot bound

A trail that ends at BOOTSTRAP_PERM_AWAIT is a create waiting on a slow-but-healthy probe.
Every capture so far was ambiguous at precisely this point, which is why three passes could not name
the mechanism.

Alongside it, .DefaultIfEmpty(null) + an explicit empty arm: a probe that completes without a
verdict now fails closed with the same verdict the faulted-fold arm already produces, and logs
loudly. It is inert today by the analysis above, so it cannot regress anything — it removes a
silent-hang class rather than changing a live behaviour.

DebuggingMessageFlow gains the general rule, since it is not specific to this call site: a chain
guarded only by Timeout still has an unhandled terminal case, and "still running" vs "terminated
empty" need different evidence and different fixes.

Still `Refs`, not `Fixes`

Unchanged and deliberate. What remains unexplained: why the probe had not answered within
0.7–3.2 s on those runs. The reframe says that may be ordinary cold-start latency rather than a
defect, but "may be" is not a root cause, and I have not reproduced the captured signature. The next
occurrence carrying these stages distinguishes slow from terminated on sight.

Also ruled out earlier in this PR and unchanged: the try-then-claim decline branch is unreachable
under AddInMemoryPersistence() (the decorator stack over InMemoryStorageAdapter, which never
emits null), so #1050's first fix is a real adjacent defect and provably not these captures.

Recurred on main ~4 h after this was closed — reopening

Run 31363873251, main df1ed76e1, shard 2, PandasExplorerLayoutAreaTest.Explorer_NoPythonNode_RendersToolbarAndDegradesGridWithoutHanging. The test BODY passed; DisposeAsync threw:

Hub mesh/fIuFzNuTjEahlmI_WY8oHw: 1 pending callback(s) after 2.00s:
  rI5qVdE90UajdNW8Xvdh2g=CreateNodeRequest@mesh/fIuFzNuTjEahlmI_WY8oHw(7328ms)
    rI5qVdE90UajdNW8Xvdh2g=CreateNodeRequest: AWAITING CreateNodeRequest→mesh/…@mesh/…(+0ms)
      → REGISTERED_AFTER_POST (Observe(delivery) overload — earlier stages not recorded)@mesh/…(+0ms)

Same signature: on-target CreateNodeRequest@mesh/<self>, past the 2 s budget, mesh alive and serving throughout. One difference worth recording — it was unanswered 5.25 s before Dispose() (registered ≈07:05:46.02, Dispose 07:05:51.277), outside the 0.7–3.2 s band noted before.

The leaked request is now identified

It is the inner create of CreateOrUpdateNodeRequest — MeshExtensions.DispatchInnerCreate — for Doc/DataMesh/PythonPandasNode/PandasExplorer/_Activity/compile-state. Proof from the same capture: at teardown the log carries

[CreateOrUpdate] inner CreateNode faulted for Doc/DataMesh/PythonPandasNode/PandasExplorer/_Activity/compile-state

which is the OnError arm of that subscription firing as the hub tore down — i.e. it was the still-pending one. And CreateNodeRequest posted by the mesh hub to itself is uniquely that call site: every other awaited CreateNodeRequest in src/ (MeshService, HubNodePersistence, ThreadSubmission, HubThreadExtensions, NodeCopyHelper, VUserHelper) already uses the pre-registering hub.Observe(request, options).

What #1050's instrumentation says

No BOOTSTRAP_PERM_* stage was recorded, on this id or anywhere in the shard. EnsurePartitionBootstrap is called with request.Id — the leaked id — and the handler's turn is scheduled inside Post, so it starts after registration and its stages would have been recorded. So this capture does not support the "slow-but-healthy permission fold" reading: the create did not reach that fold (Doc already has root + grant, so the bootstrap short-circuits before the gate).

What it cannot say is where the delivery did go — because the trail is blind by construction here, exactly as RequestFateLedger itself annotates: "re-run with the caller switched to Observe(request, options), which registers first." Track starts a trail at REGISTRATION, so POSTED / RECEIVED / ENQUEUED / ROUTED were all lost.

The defect at that call site, and why it can lose the reply

DispatchInnerCreate posted and only then registered:

var forwarded = hub.Post(inner, o => …WithTarget(hub.Address))!;
hub.Observe(forwarded).Subscribe(…);

It runs off the hub action block — it is the continuation of persistence.Read(...) — so KickDrain finds draining == false and schedules the create's turn onto turnScheduler, on another thread, before Post returns (PostImplGeneric calls ScheduleNotify synchronously).
HandleCallbacks drops a response whose correlation id has no registered subject ("No subject found for response … treating as processed"). A reply that lands in that window is consumed and gone; the callback then waits forever, and the upsert never answers.

This is the same defect the repo has already fixed at four other sites, with the rule written down in MeshNodeStreamExtensions and proven deterministically by GetMeshNode_WarmOwner_DropsResponse_WhenSubjectRegisteredAfterPost ("a WARM owning per-node hub answers in sub-millisecond time, so under thread-pool contention a preemption between Post and Observe let the reply land before the subject existed").

Fix + pin

PR to follow: DispatchInnerCreate switched to hub.Observe(request, options), plus UpsertInnerCreateObservationTest, which pins the drop deterministically on this exact message shape (self-targeted CreateNodeRequest on the mesh hub) with both orderings side by side and no sleep — the fence is node readability plus two round-trips.

Honest limit: the deterministic pin proves the mechanism, and identifies this call site as the only one that could produce a CreateNodeRequest@mesh/<self> leak — it does not prove this particular CI capture went through that window, and it cannot, because the post-hoc registration is what erased the evidence. If the leak recurs after this lands, the trail will now carry every stage and will name the mechanism outright. That is the point of the change as much as the fix is.

Closing — this time on a named mechanism plus qualifying runs, not on absence of evidence.

What it actually was

The leaked request was identified: CreateOrUpdateNodeRequest's inner create (MeshExtensions.DispatchInnerCreate). It posted first and registered the response subject second, and HandleCallbacks drops a response whose correlation id has no registered subject — so a reply landing in that window is gone and the upsert never answers.

The window is real because DispatchInnerCreate runs off the action block (it is the persistence.Read continuation), so KickDrain schedules the create's turn onto another thread before Post returns. Every other awaited CreateNodeRequest in src/ already used the pre-registering overload; this was the one call site that could produce the shape.

Fixed in PR #1083, together with the reason it kept coming back: AsynchronousCalls.md presented the two overloads as interchangeable and carried a worked example headed "When you already have a delivery" that was the defect verbatim. That bug had been written five times.

Evidence for the close

#1083 merged 2026-08-10T08:20Z. Since then, eight main runs with all six Run tests shards genuinely executed (none skipped — checked per-job, not just the suite conclusion) and zero occurrences of PandasExplorerLayoutAreaTest or any sibling member.

Why this close differs from the last one

This issue was closed once before on the reading that the captures were a slow-but-healthy permission fold — the 15 s bound versus the 2 s quiescing budget — i.e. possibly no leak at all. That was wrong, and it recurred ~4 h later. The later capture also refuted it directly: the BOOTSTRAP_PERM_* instrumentation recorded no stage on the leaked id, and Doc already has root + grant so the bootstrap short-circuits before that gate.

Worth keeping for the record: 26 of 30 local DISPOSE_QUIESCE_LEAK records were separately proven to be the detector racing its own budget, so that family was noisier than it looked — but the defect underneath was real.

Reopen if it recurs. The RequestFateLedger trail (PR #1011) and the BOOTSTRAP_PERM_* stages will name the mechanism on sight rather than leaving a fourth ambiguous capture.

fix(#2423): the test harness stops driving the mesh through the ROUTER

Callbacks still pending past the teardown quiescing budget (FutuRe EuropeRe, Cession, PandasExplorer) — a CreateNodeRequest the mesh hub never answers #981

Description

Summary

Why this is a real defect, not test noise

Suggested approach

Activity

rbuergi commented on Aug 9, 2026

rbuergi commented on Aug 9, 2026

Re-measured on current main: still live after #996, and it is not #613

#996 did not fix it — and the family is wider than FutuRe

The two captured instances share a precise signature

Corrections to this issue's own framing

Still open, and what I did not do

rbuergi commented on Aug 9, 2026

rbuergi commented on Aug 9, 2026

A third capture — and it narrows the root cause to the parent's stuck request

The child's stuck request is a consequence, not the defect — the arithmetic says so

What that leaves as the actual question

The one instrument that would end this

rbuergi commented on Aug 9, 2026

Narrowing, from the captures themselves — two hypotheses dead, one mechanism confirmed in code

rbuergi commented on Aug 9, 2026

The failure now reports the HANDLER side — PR #1011

What was missing

The binary question it was built to settle

One code-reading observation — a candidate, NOT a proven cause

What it rules in and out

Where the evidence lands

Also corrected

rbuergi commented on Aug 9, 2026

Two things the local trace turned up while validating the channel

rbuergi commented on Aug 9, 2026

Not the same defect as #1029 — and here is the proof, plus what it rules out here

What #1029 turned out to be

What that means for the RequestFateLedger verdicts to expect here

One thing I can add to that candidate: it is REACHABLE, not just a code reading

What I could not reproduce

rbuergi commented on Aug 9, 2026

The empty-completion lead, stated as a defect in its own right — and exactly what capture confirms it

It is a defect whether or not it is THIS issue's captured failure

What capture confirms it — stated concretely so the next occurrence is not wasted

Still not fixing it here

Unchanged

rbuergi commented on Aug 9, 2026

The empty-completion lead is now fixed as a defect in its own right — and it is provably NOT this issue's captures

What shipped

Why this is not this issue's captured failure

What the fix does do for this issue

One new lead, from reading the Monolith wiring

Unchanged

rbuergi commented on Aug 10, 2026

The TakeDecisionOutsideGate lead: RULED OUT — and ruling it out reframes the issue

The structural worry was real

Why it cannot fire

What that reframes about this issue

The instrument that settles it — PR #1050 (branch updated)

Still Refs, not Fixes

rbuergi commented on Aug 10, 2026

Recurred on main ~4 h after this was closed — reopening

The leaked request is now identified

What #1050's instrumentation says

The defect at that call site, and why it can lose the reply

Fix + pin

rbuergi commented on Aug 10, 2026

What it actually was

Evidence for the close

Why this close differs from the last one

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Re-measured on current `main`: still live after #996, and it is not #613

What that means for the `RequestFateLedger` verdicts to expect here

The `TakeDecisionOutsideGate` lead: RULED OUT — and ruling it out reframes the issue

Still `Refs`, not `Fixes`