A hub past [QUIESCE-OK] still accepts new correlated requests — the intake gate opens one phase too late #3506

A hub past [QUIESCE-OK] still ACCEPTS new correlated requests, and they can never be answered

Split out of #3499, which measured this shape but was wedged by a different one. #3499's fix does
NOT cover this; nothing does.

The defect

MessageService.ScheduleNotify's shutdown gate refuses new deliveries only from
MessageHubRunLevel.DisposeHostedHubs onward:

if (hub.RunLevel >= MessageHubRunLevel.DisposeHostedHubs
    && delivery.Message is not ShutdownRequest and not DisposeRequest)

The Quiescing phase is therefore wide open. A hub that has entered [QUIESCE-START] — and even
one that has already logged [QUIESCE-OK], having drained everything it owed — keeps accepting
inbound deliveries, keeps handling them, and keeps registering response callbacks for requests it
POSTS. #3261 measured it creating hosted hubs in that window too:

11.928 [QUIESCE-START] TestData/dnt…: 0 pending callbacks at dispose entry: <none>
11.928 [QUIESCE-OK]    TestData/dnt…: drained 0 callback(s) in 0ms
11.968 Start processing SubscribeRequest (ID: p2qsFux3…, cache/iIN4… -> TestData/dnt…)
11.969 Creating new hosted hub for address sync/4wzOafy… in host TestData/dnt…

Work taken on after the drain has, by construction, no drain left to finish it: the quiesce budget
has already been spent, so the next phase cancels it.

Measurement (2026-09-06, three independent bake runs)

Counted from the RECEIVED runLevel=<level>@<issuing hub> stage in the request-fate trails printed
by [QUIESCE-TIMEOUT]. Six of nine pending callbacks were registered by a hub that was ALREADY
Quiescing:

Each one costs the full 2 s quiesce budget, ends in forcibly cancelling, and hands the requester a
HubDisposedBeforeResponseException.

Why this is filed separately rather than fixed with #3499

It is benign in every occurrence measured so far, and it is not what wedges an install. The
requester DOES get a typed, transient answer, and eight packages carrying this shape passed the gate
on run #7937 alone. The wedge #3499 fixed is the other row of the table: a request accepted while the
hub was Started, whose owner-side patch had already committed and whose teardown fault was
classified as the terminal Unknown instead of the retryable OwnerDisposing
(#3505).

That said, forcibly cancelling is the tell of an unfixed root, and this is the root: teardown lets
accepted work finish; it must not keep accepting new work it cannot finish.

🚨 Not the fix

Widening the 2 s QuiesceTimeout. #3261 settled that already: a bigger budget converts a leaked
callback into a slower leaked callback. The same goes for re-arming it further
(MaxQuiesceRearms exists for a different case — a reply owed by a sibling in this mesh that is
itself shutting down and WILL answer).

Acceptance

• A hub that has entered Quiescing does not take on new correlated work it cannot answer: the
  intake gate refuses from the first instant of disposal, not from DisposeHostedHubs, with the same
  transient ErrorType.ShuttingDown NACK the later gate already posts (NackThroughParent, activation
  identity and all).
• The exemptions the later gate needs are preserved — ShutdownRequest / DisposeRequest, and
  whatever else teardown itself must route — plus, deliberately, any delivery that settles work
  already accepted (a response correlating to a pending callback), which is exactly what the
  quiesce drain is waiting for and must not be refused.
• A deterministic pin that goes RED on today's code. [QUIESCE-OK] followed by an intake is a
  reproducible sequence (A hosted hub's leaked callback is reported on its owner, and a subtree drains under one Quiescing budget #3261's DanglingNodeTypeUpdateTest with MESHWEAVER_MSG_TRACE=1 and
  MeshWeaver.Messaging: Debug produced it), and after fix(#3499): a teardown fault is a RETRYABLE verdict, not a terminal one — an install survives its own root recycling #3505 the tester image
  logs [QUIESCE-START] so a bake occurrence is attributable in one read.
• No bound is widened.

Provenance

Measured while root-causing #3499; the [QUIESCE-OK]-then-late-intake trace is #3261's.