MeshNodeStreamCache permanently poisons a path after one failed subscribe — same cached TimeoutException replayed forever (memex-cloud, ci.2926) #1202

What is failing

On the live memex-cloud portal (https://memex.meshweaver.cloud), a point read of certain node paths never succeeds — it fails instantly by replaying a cached TimeoutException from an earlier failed subscribe, rather than re-probing the owner. The path is poisoned for the process lifetime.

Reproduced live via MCP against the deployed portal on 2026-08-11, 07:44–07:56Z:

get @DataModeling/Formatting
→ TimeoutException: No response received in hub cache/gmIIqhF220CKKxqxgY_J7Q within 00:01:00
  for request SubscribeRequest (id=3kjIJ9lXPUCqPU5Zkgmhjw) → target DataModeling/Formatting.

The decisive observation: three reads spanning ~11 minutes all returned the byte-identical request id 3kjIJ9lXPUCqPU5Zkgmhjw. A genuinely re-attempted subscribe would allocate a fresh request id every time. The same exception instance is being replayed, so after the first failure no further SubscribeRequest is ever issued for that path.

StormMaxCooldown is 5 minutes, so within an 11-minute window the storm-breaker should have re-probed at least once and produced a new id. It did not.

Scope — this is NOT "a user session went away"

So storage, the query providers and the pod itself are all healthy. The node demonstrably exists (it is returned by the children listing and by search). Only the per-node hub subscribe path through the shared MeshNodeStreamCache is broken, and only for the paths that got poisoned.

Affected paths cut across partitions (DataModeling, rsalzmann) while other partitions (Admin, Doc) are fine — so it tracks which paths happened to fail a subscribe once, not which partition is unhealthy.

Why the existing issues do not cover this

• NumberFieldView subscribe to user hub times out — downstream of hub cache unavailability #1195 / NumberFieldView subscribe to user hub times out — same hub-cache routing failure as issue #1195 #1196 / TextFieldView subscription to user hub times out when target session is gone #1197 (all OPEN) record the same symptom through the same hub cache id gmIIqhF220CKKxqxgY_J7Q, but triage them as "user sglauser's session had ended", severity Low, blast radius one user. That scoping is wrong: the same cache is poisoning ordinary content paths for every reader, and it does not heal.

• MessageHubGrain activation frequently times out during initialization due to unresolvable mesh nodes #1186 (CLOSED) was the named root cause, closed as fixed by Grain activation no longer times out waiting on itself #1062 / A message arriving mid-activation could overtake the queue that ordered it (fixes the kernel CS0103 reorder) #1155 / The boot plugin install no longer races the NodeType bake it depends on #1161 / An in-place update no longer costs the router a storage query per message #1179 with an explicit reopen condition:

  "If this fingerprint recurs on a pod running ci.2877 or later, that would be new evidence and worth a fresh look."

  That condition is now met. Admin/PlatformVersion reports 3.0.0-rc1.ci.2926+build.639220298296349729.9fd339b71d72d932190e36c4a86955c1fe50a351, self-updated at 07:39:13Z, and the failures above are from 07:44–07:56Z — after the roll.

Crucially, none of them document the stickiness, which is the actual defect here. A transient owner timeout is survivable; a transient owner timeout that permanently poisons the path is not.

Root cause hypothesis — the Reprobing latch can never be cleared once it is set spuriously

High confidence; all line refs src/MeshWeaver.Hosting/MeshNodeStreamCache.cs.

1. CreateEntry subscribes the upstream handle straight into a ReplaySubject<MeshNode>(1) (~L584–L618). When the upstream SubscribeRequest times out at 60s, OnError lands on that subject, which is now terminally faulted — it replays that exact exception to every future subscriber, forever.
2. The faulted Entry stays in _streams. GetEntry (~L522) hands it back on every subsequent read because TryTouch() only checks evicted, not faulted.
3. The only escape is the storm-breaker branch in GetStreamRaw (~L862–L882), which evicts the stale entry so the next read re-probes. It is guarded by a one-way latch:

   if (!neg.Reprobing
       && _negative.TryUpdate(path, neg with { Reprobing = true }, neg)   // ← latch set here
       && _streams.TryRemove(path, out var staleLazy) && staleLazy.IsValueCreated)

   TryUpdate sets Reprobing = true as a side effect inside the && chain. If either later conjunct fails (_streams.TryRemove losing a race with the idle sweep or a concurrent GetEntry, or IsValueCreated == false), the latch is set but the poisoned entry is never evicted.
4. Reprobing is reset in exactly one place — RecordNegative (~L789–L796) — and RecordNegative is invoked only from the bookkeeping observer on a real upstream subscription's OnError (~L620). Once step 3 leaves the poisoned entry in place, every later read is served the replayed error from the faulted ReplaySubject; no new upstream is opened, so RecordNegative never runs again, so Reprobing stays true forever.

Net: !neg.Reprobing is permanently false → the eviction branch is dead → the faulted entry is pinned → every reader gets the original 60s TimeoutException with the original request id. Which is exactly what is observed.

The one remaining exit is the idle sweep (ReleaseIdleReadStreams, ~L700), but it needs ReadStreamIdleExpiration (10 min default) with no subscriber and no read/write hit — and every read calls TryTouch(), which refreshes the sliding idle window. So an actively-viewed page (a user refreshing a broken lesson) keeps resetting the timer and keeps itself poisoned. The failure is self-sustaining under exactly the load pattern that surfaces it.

Impact

High, and mis-triaged as Low. Any node whose owner hub is slow once — during a rollout, a cold activation, a routing blip — is permanently unreadable through the shared cache for the lifetime of that pod. The GUI symptom is a view that never loads (this is the NumberFieldView/TextFieldView/ThreadChatView family in #1195/#1196/#1197/#1175). Because MeshNodeStreamCache is process-wide infrastructure serving routing, NodeType activation and path resolution, poisoned paths degrade far more than the page the user is looking at. It survives self-update only in the sense that a fresh pod starts clean and then re-accumulates.

Suggested direction (root cause, not a bound)

• A faulted entry must never be servable from _streams. Treat terminal error as an eviction trigger at the point the fault is observed (the bookkeeping observer already sees it) rather than leaving it for a latch-guarded branch to clean up later.
• Reprobing should not be settable without the eviction actually happening — hoist the TryUpdate out of the && chain, or give the latch its own expiry so a spurious set cannot be permanent. A latch whose only reset lives on a code path the latch itself prevents from running is unrecoverable by construction.
• Worth a deterministic pin: first subscribe errors → Nth read opens a genuinely NEW upstream SubscribeRequest (assert the request id changes), and a spuriously-set Reprobing still recovers.

Please raise ReadStreamIdleExpiration-based recovery expectations only as a fallback, not as the fix — the sliding-window refresh means it does not fire under load.

Repro

Against the live portal, no special state needed:

get @DataModeling/Formatting     # → TimeoutException, note the request id
# wait > 5 min (StormMaxCooldown), then:
get @DataModeling/Formatting     # → same exception, SAME request id  ← the bug
get @DataModeling/*              # → succeeds, proves the node exists