A reply routing cannot deliver goes to the waiter, not the floor — and a quiescing hub stops taking work it can never answer by rbuergi · Pull Request #3302 · Systemorph/MeshWeaver

rbuergi · 2026-09-04T13:22:54Z

Why — this is currently reddening every open PR

MeshWeaver.Graph.Test.NackReachesTheWaiterDuringTeardownTest.OwnerDisposingUnderMeshTeardown_StillAnswersTheWaitingCaller is failing shard 4 on #3297, #3299 and #3300 simultaneously, and failed core CI runs 33861078925 and 33865385033 earlier today. It is not three problems and it is not a flake to retry — it is one broken edge, and the test's own log names it:

[fence] caller is armed on the late watch (ArmedCount=1)
[dispose] mesh disposal invoked — parent is past DisposeHostedHubs
Message delivery failed for PatchDataResponse (ID: l9ret6X…) in TestData/teardown-nack-node:
  Hub TestData/teardown-nack-node cannot route PatchDataResponse to cache/VLe3ZpQ…

The owner answered correctly. Its post was accepted — its own run level is still open, so PostImplGeneric does not refuse — and one turn later HierarchicalRouting finds the parent past DisposeHostedHubs and drops the reply: classified, logged, and told to nobody. The waiter is in the same process, with its registry entry armed and able to take the answer directly, but the routing layer had no way to reach it. So the caller burns its whole budget in silence.

What changes

1. An undeliverable correlated reply is handed over, not dropped. New IUndeliverableReplySink in MeshWeaver.Messaging.Hub; HierarchicalRouting offers a dropped reply to a local waiter on the two arms where it gives up (parent past DisposeHostedHubs, and the disposing no-route arm). LatePatchResponseRegistry implements it, registered as a factory over the same singleton — a second instance would dispatch into a registry nobody armed, which reads exactly like "nobody was waiting". A sink that throws cannot turn a classified drop into a routing fault.

🚨 Only where the delivery is being dropped. Serving an armed caller alongside a healthy post was tried twice at the patch-verdict seam and reverted both times — the sink dispatches synchronously on the owner's turn, ahead of the state the ack is about (ComboGateRollTest run 33863349195, ImportTypeBeforeInstanceTest run 33865385033). Here there is no post left to race, which is what makes it safe.

2. A quiescing hub refuses new work it can never answer. Quiescing exists so the replies a hub is still owed can come in, and the intake gate only refused from DisposeHostedHubs on. Between the two, a hub whose quiesce had finished still admitted brand-new work — and built hosted hubs to serve it. Measured on #3261's branch (DanglingNodeTypeUpdateTest):

[QUIESCE-START] TestData/dnt…: 0 pending callbacks at dispose entry: <none>
[QUIESCE-OK]    TestData/dnt…: drained 0 callback(s) in 0ms
Start processing SubscribeRequest (cache/… -> TestData/dnt…)          ← 40 ms later
Creating new hosted hub for address sync/4wzOafy… in host TestData/dnt…

Now refused at the door with the same transient ShuttingDown NACK the DisposeHostedHubs arm mints. Responses stay admitted — the RequestId is read off the envelope, so a cross-hub RawJson still counts — and so does lifecycle traffic.

🚨 Scoped to what is addressed here, by somebody else. ScheduleNotify is not an inbox: a hub's own outbound posts pass through it too. The first version refused those and SourcesWatcherStopsAtTeardownTest went red — the defect in miniature, since NackThroughParent declines to answer a hub's own delivery and the callback it had just armed then held Quiescing for the whole budget.

Tests

QuiescingRefusesNewRequestsTest: a request arriving after Quiescing began is refused at the door (the victim genuinely owes a withheld reply, so the phase is really waiting, and the late request's handler would answer — a pass can only come from the refusal), plus an anti-vacuity anchor that a reply the hub is owed is still admitted. Verified by reverting the production change: the first fails with "No exception was thrown", the anchor passes either way.
DisposalRaceNackTest door one now observes the acceptance it always assumed, waiting on the fate ledger for ENQUEUED@<victim> before disposing. It had been decided by whether the host's routing turn beat Dispose(); adding a class to the suite shifted the scheduling and it lost. (First attempt at that wait used o.WithMessageId(...), which Observe<TResponse> overrides — so it polled a key that never existed and timed out having proved nothing. The Observe(request, options, messageId) overload is the right one.)

Verification

Release, DOTNET_PROCESSOR_COUNT=2 (CI's shape): Graph 1324/1324 (including the failing test above), Hosting 350/350, Data 475/475, Memex.Portal.Shared 727/727, DisposalRaceNackTest + QuiescingRefusesNewRequestsTest 4/4.

What this does NOT do

It does not close #3261. I re-measured its eight reds on a tree carrying this change: still 7 of 7 failing (Graph 1, Compiler.Pipeline 2, Portal.Shared 4). The graph specimen's ledger verdict moved — from "a reply WAS posted … lost" to "no handler was ever entered" — so the reply-loss path it took is closed and a different leak now dominates. The remaining shapes, counted off the ledger: "a reply WAS posted … still pending" ×6, "no handler was ever entered" ×3 (a request that cannot be routed during disposal, classified and still unanswered — a third hole, on the request leg), "handler entered, nothing recorded since" ×2, "chain FAULTED, no reply" ×1. Those need a further change set; this one is scoped to the edge that is breaking CI today.

Pairs-with: none — one new public interface, no removals.

🤖 Generated with Claude Code

…ting cannot deliver is handed to the waiter instead of dropped Two halves of one defect: work that arrives at a hub too late to be served, and an answer that cannot get back out. Both end as a caller holding a pending callback until its own budget expires, which is what #3261's detector measures — 72 of 690 disposals at ≈2.0 s, silently. **1. Quiescing refuses new requests (ScheduleNotify).** Quiescing exists so the replies a hub is still OWED can come in, and the intake gate only refused from DisposeHostedHubs on. Between the two, a hub whose quiesce had already finished still admitted brand-new work — and built hosted hubs to serve it. Measured on #3261's branch (DanglingNodeTypeUpdateTest, 2026-09-04): [QUIESCE-START] TestData/dnt…: 0 pending callbacks at dispose entry: <none> [QUIESCE-OK] TestData/dnt…: drained 0 callback(s) in 0ms Start processing SubscribeRequest (cache/… -> TestData/dnt…) ← 40 ms later Creating new hosted hub for address sync/4wzOafy… in host TestData/dnt… Three late SubscribeRequests, fresh sync/* children built for them, on a hub already being torn down. One was answered; the others died with it. Now such a delivery is refused at the door with the same transient ShuttingDown NACK the DisposeHostedHubs arm mints — an answer in milliseconds instead of a wait. Responses are exempt (the RequestId is read off the ENVELOPE, so a cross-hub RawJson still counts), and so is lifecycle traffic. 🚨 Scoped to what is addressed HERE by somebody else. ScheduleNotify is not an inbox: a hub's own OUTBOUND posts pass through it too. The first version refused those, SourcesWatcherStopsAtTeardown went red, and the reason is the defect itself in miniature — NackThroughParent declines to answer a hub's own delivery, so the callback it had just armed held Quiescing for the whole budget. **2. An undeliverable reply is handed over, not dropped (HierarchicalRouting).** The owner answers, its post is accepted (its run level is still open), and one turn later routing finds the parent past DisposeHostedHubs and drops it — classified, logged, told to nobody: Message delivery failed for PatchDataResponse (ID: l9ret6X…) in TestData/teardown-nack-node: Hub … cannot route PatchDataResponse to cache/VLe3ZpQ… while the waiter sits in the SAME process with its registry entry armed. That is NackReachesTheWaiterDuringTeardownTest, red on CI runs 33861078925 and 33865385033 and locally under load. New IUndeliverableReplySink: routing offers a dropped CORRELATED reply to a local waiter before discarding it; LatePatchResponseRegistry implements it, registered as a factory over the same singleton (a second instance would dispatch into a registry nobody armed). 🚨 Only where the delivery is being DROPPED. Serving an armed caller alongside a healthy post was tried twice at the patch-verdict seam and reverted both times — the sink dispatches synchronously on the owner's turn, ahead of the state the ack is about (ComboGateRollTest, ImportTypeBeforeInstanceTest). Here there is no post left to race. **Tests.** QuiescingRefusesNewRequestsTest: a request arriving after Quiescing began is refused at the door (the victim genuinely owes a withheld reply, so the phase is really waiting, and the late request's handler WOULD answer — a pass can only come from the refusal); plus the anti-vacuity anchor that a reply the hub is owed is still admitted. Verified by reverting the production change: the first fails ("No exception was thrown"), the anchor passes either way. DisposalRaceNackTest door one now OBSERVES the acceptance it always assumed — it waits on the fate ledger for ENQUEUED@<victim> before disposing. `Observe` posts on the HOST, whose turn then routes down to the victim, so whether the request got in was decided by which turn won; adding a class to the suite shifted the scheduling and it lost, failing with the DisposeHostedHubs arm's own NACK. Verified in Release at DOTNET_PROCESSOR_COUNT=2: Graph 1324/1324, Hosting 350/350, Data 475/475, Memex.Portal.Shared 727/727, Messaging.Hub (in progress at commit time; reported on the PR). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

…polls the real key The first version of the wait set the correlation id via `o.WithMessageId(...)`, but `Observe<TResponse>` mints its own — so the poll asked the ledger for a key that never existed and timed out at 36 s having proved nothing. A wait that cannot succeed is the same defect class as a gate that skips: it settles for reasons unrelated to its subject. `Observe(request, options, messageId)` is the framework's overload for exactly this. Both doors now pass in 618 ms, and the wait observes the acceptance the test always assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

github-actions · 2026-09-04T13:27:32Z