fix(routing): the pod-hub leg classifies a deactivated activation instead of defaulting to terminal (#2299)

src/MeshWeaver.Documentation/Data/Architecture.md

@@ -378,6 +378,7 @@ Each theme starts with its introductory page, followed by related architecture t
 - [A Name That Does Not Resolve Is Not Transient](ANameThatDoesNotResolveIsNotTransient) — the default pipeline retried a hostname that does not exist three times and logged each attempt at Error, so a URL in somebody's data manufactured a platform incident; the one socket error that is permanent, why the breaker must not count it either, and the control that keeps a nameserver hiccup retryable
 - [A Departed Silo Is Not a Delivery Defect](ADepartedSiloIsNotADeliveryDefect) — two incidents (191 and 3,959 occurrences) read as two defects for four weeks and are ONE root; which predicate sees which of the four rejection shapes, why the cure is the classifier and never a retry, and why an incident fingerprinted on a dependency's logger counts attempts rather than verdicts
 - [A Timed-Out Delivery Is Still Held by the Callee](ATimedOutDeliveryIsStillHeldByTheCallee) — a response timeout is a caller-side give-up timer, so the six-retry ladder sized for an instant rejection re-sent seven copies of every slow delivery and held a dispatch slot for 3 m 40 s; the three-predicate ladder that separates "is this transient" from "may we send it again", and the one caller that keeps the wider answer because it is idempotent
+- [The Subsystem in the Title Is Not the Cause](TheSubsystemInTheTitleIsNotTheCause) — six tickets named two Orleans subsystems and grouping them on that was wrong three times; the three questions to ask per ticket, the pod-hub address suffix that is a process id rather than an operation id (which moved one investigation to the other side of the call), the two `memory-streams` tickets whose live traffic has different owners, and the precondition for folding a symptom into its root
 - [Reading a Routing Saturation Report](ReadingARoutingSaturationReport) — the `[ROUTE]` back-pressure line is a gauge over three different facts, and for most of the tickets it produces the causal arrow runs the other way; which number answers which question, the two opposite pool states that printed the same number because the report showed the one gauge blind to the cause it named, and the oldest-leg age that separates load from a leaked slot in a single sample
 - [A Bulk Create Compensates Per Node](BulkCreateCompensation) — every row is durable before any post-creation handler runs, so one critical failure left the failed node AND every node after it, whose handlers never ran and which nothing can tell apart from a success; what the rollback removes, why it walks backwards, and the measured reason the stop is a fault and not a `Take(1)`
 - [Undetermined Is Not No](UndeterminedIsNotNo) — a read that did not answer is a THIRD state; the second door that shared the first door's failure domain, and the rule for what a gate does with "I could not determine"

src/MeshWeaver.Documentation/Data/Architecture/ADepartedSiloIsNotADeliveryDefect.md

@@ -58,7 +58,7 @@ IsScopeTeardown`, and `Failed` for everything else. What production actually del
 | `…is not stable to perform the lookup… Retry later.` / `hop limit is reached` / `on invalid silo` | the grain directory is mid-handoff | `IsDirectoryUnstable` (#1742 / #2357 / #3139) |
 | `Unable to connect to S10.244.2.223:11111:148812047 …` — `HostUnreachable`, `ConnectionRefused` | nothing is listening at that pod incarnation | `IsDepartedSiloRejection` |
 | `The target silo is no longer active: target was …:146524552, but this silo is …:146534005` | the pod restarted and reclaimed its address | `IsDepartedSiloRejection` |
-| `…for 2 times after "DeactivateOnIdle was called." to invalid activation. Rejecting now.` | the target **grain** deactivated while the message was in flight | its own predicate — a separate change |
+| `…for 2 times after "DeactivateOnIdle was called." to invalid activation. Rejecting now.` | the target **grain** deactivated while the message was in flight | `IsDeactivatedActivation`, gated on a discriminator — and for one release that gate made it unreachable on the leg that produces it; see below |
 
 The first row was already recognised. The middle two are this page's subject. The fourth is a
 different statement — about an activation on a live silo, not about the silo — and is handled
@@ -74,6 +74,105 @@ one already covers — `IsTransientFailure`'s `OrleansMessageRejectionException`
 shapes, so they had been retried with a fresh resolve six times since PR #2314. Only the verdict was
 wrong.
 
+## 🚨 The fifth instance of that shape: a SAFE DEFAULT made the fourth row's predicate inert
+
+The shape above has a variant that is harder to see, because the classifier *can* read its input —
+it was simply never handed it.
+
+`IsDeactivatedActivation` is deliberately not a bare text match. The same rejection means two
+opposite things, and `GrainActivationFailureRegistry` documents the other one: a **per-node hub** in a
+*persistent* activation-fault loop — a NodeType whose compile cannot materialise a hub configuration
+— has an alive window of about zero, so every delivery lands in a deactivation window and Orleans
+answers with this exact sentence. That grain never recovers, and demoting it would hide a real defect
+behind a transient NACK. So the text is a NECESSARY condition and the registry is the discriminator:
+
+```csharp
+internal static bool IsDeactivatedActivation(Exception ex, bool activationErrorRecorded) =>
+    !activationErrorRecorded && /* the text test */;
+
+internal static ErrorType ClassifyDeliveryException(
+    Exception ex, Func<bool>? scopeDisposed = null, bool activationErrorRecorded = true) => …;
+```
+
+The default is `true` — *assume the worse case* — so a caller that cannot consult the registry leaves
+the verdict terminal, exactly as this page's own rule demands.
+
+**And the pod-hub leg was such a caller.** `BuildPodHubRoute`'s terminal arm called
+`ClassifyDeliveryException(ex, IsServiceScopeDisposed)` — two arguments — so on that leg the arm could
+never fire, while `BuildGrainRoute` passed
+`activationErrorRecorded: !string.IsNullOrEmpty(activationError)` and worked. The evidence says which
+leg matters: the rejection names `IPodHubGrain.Deliver`, and #2299 is almost entirely this shape — 947
+recorded occurrences, every one of the newest samples, each logged as *"surfacing **Failed**
+DeliveryFailure to sender …"*. The predicate shipped, its unit facts were green, and production kept
+printing the pre-fix verdict in one word.
+
+> 🚨 **A predicate gated on a discriminator that defaults to the safe answer is INERT at every call
+> site that does not pass the discriminator — and the site that produces the fault may be one of
+> them.** There is nothing to grep for, because the defect is an argument that was not written: the
+> predicate reads correct, and the call site compiles.
+
+### Why `false` is a FACT on this leg, not a guess
+
+Passing it on a leg that merely *lacks* a registry would be a guess. Here it is a statement about the
+grain:
+
+1. **The registry is per-node-hub by contract** — "the LAST activation failure observed for each
+   per-node-hub grain" — and only `MessageHubGrain` writes it.
+2. **A `PodHubGrain` has nothing to fail at activation**: no NodeType, no hub configuration to
+   materialise, and a class contract that it must *not* throw from `OnActivateAsync` because the
+   refusal is the CALL's answer (`PodHubNotHereException`). It cannot enter the loop the discriminator
+   separates, so it never records an error and never could.
+3. **Therefore the only way a pod-hub activation becomes invalid is the idle deactivation it requested
+   of itself** when it answered "not here" — a lifecycle transition by construction, which is this
+   page's bar.
+
+`ClassifyPodHubDeliveryException` states that once, with the argument attached, and the leg calls it.
+
+### What the corrected verdict does NOT fix
+
+The **bounce that produces the rejection**. With `[PreferLocalPlacement]`, a delivery to a pod-hub
+address with no live owner activation is placed on the *caller's* silo, which holds no local route for
+it; `PodHubGrain.Deliver` then requests its own idle deactivation **and** throws. The throw answers
+*that* message terminally; every message arriving in the deactivation window is answered by **Orleans**
+instead, transiently, and the router's six re-resolving retries can each place another throw-away
+activation whose own window bounces the next arrivals. That is self-sustaining, and it is a lifecycle
+question rather than a classification one:
+
+| candidate | what it costs |
+|---|---|
+| stop requesting deactivation in `Deliver`, so the grain's own TERMINAL refusal is the only answer | the throw-away activation lingers instead of being reaped promptly. It holds no route and no stream, and the claim path still converges because `Attach` requests deactivation on its own non-owning path — but the ceiling becomes *silos × addresses recently mis-routed to them* rather than zero |
+| bound that ceiling with Orleans' own reaper — a collection-age limit on the grain type, which can only affect UN-pinned activations (the owner's is pinned indefinitely, a released address carries a finite tombstone) | the age must stay at or above the collection quantum; it is a bound stated from source rather than a timer anyone wrote |
+| refuse to re-send the delivery on this rejection for this leg alone, since prefer-local guarantees the retry re-places the squatter | loses the retries that DO succeed because the owner's claim landed between attempts — a reduction in a symptom's volume, not a removal of its cause. Recorded so it is not mistaken for one |
+
+The first two belong together and want one measurement on a real cluster: is an inert activation cheap
+enough that idle collection alone is an acceptable reaper. None of the three is a classification
+change, which is why the classification landed on its own.
+
+### What is pinned, and what is not
+
+`PodHubDeactivatedActivationClassificationTest` pins the **decision** from both sides, with both
+production texts quoted verbatim: the pod-hub classifier answers `ShuttingDown`, the general
+classifier with its own defaults still answers `Failed` for the identical exception, and the verdict
+flips on `activationErrorRecorded` alone — so the discriminator cannot be collapsed into an
+unconditional text match. It also pins that the container probe is forwarded and that a rejection
+carrying no recognised phrase stays terminal on this leg too.
+
+🚨 **And it pins the WIRING, which took a second pass.** The first version of this change pinned only
+the two classifiers, and review caught the obvious consequence: reverting the one line in
+`BuildPodHubRoute` that chooses between them left all eight facts green — **the same shape as the
+defect being fixed**, an argument not written at a call site, one level out. So the arm is now one
+tested function: `TerminalCallFailure` delegates to `AnswerPodHubCallFailure`, which holds both
+decisions (which classifier, and the level the verdict deserves), and two facts drive that function
+and capture what it hands the sender and the logger.
+
+Measured, with the revert applied as a control: **2 of the 10 facts go red and 8 stay green** — which
+is exactly the blindness the review named, now visible from inside the suite rather than only in
+prose. The remaining unpinned surface is a one-line delegation with no logic in it.
+
+The transferable half: **when the defect is "a decision was not expressed at a call site", a fix
+pinned only by facts about the decision's INPUTS reproduces it.** The fix has to move the decision
+somewhere a test can execute the same code production does.
+
 ## The bar, and why a timeout deliberately fails it
 
 `ClassifyDeliveryException` is **narrower than `IsTransientFailure` on purpose**, and the asymmetry is