ThreadChatView times out subscribing to ThreadComposer hub for user session #1175

Activity

1. rbuergi commented on Aug 10, 2026

   

   rbuergi

   on Aug 10, 2026

   Contributor

   More actions

   Triage verdict: not a new mechanism — two known layers, both already in flight (PR #1177 client-side, #1172 server-side).

   Server side — why the subscribe timed out at that moment. Same window as #1174 (full timeline + live reconfirmation there). The 13:38:56Z self-update roll's pod memex-portal-deployment-59479dc7d9-kmwvr joined the Orleans cluster at ~13:39 but became Ready only at 15:59:37 (Ready.lastTransitionTime) — 2 h 20 min held in the nodetype_bake startup gate while it reran the full dynamic-NodeType rebuild. All 4 occurrences (14:13:34 → 14:20:09, pod …6bd9c46f8b-z9t6r, user rsalzmann — the same user/pod/burst as #1174's 14:13–14:20 tail) lie inside that join-to-Ready window: the rsalzmann/_Thread/ThreadComposer per-node-hub activation landed on the mid-bake silo, where #1172's route-slot exhaustion behind non-terminating _Activity/compile dispatches starves activations and deliveries, so the 60 s SubscribeRequest budget in the hub cache expired. Root cause of that starvation = #1172; exposure already cut ~44× by merged PR #1173 (these incidents predate its merge by ~3 h, under the old ~20-min roll cadence).

   Client side — the actual defect in ThreadChatView, and it is already fixed in an open PR. ThreadChatView.OpenComposerProjection (and ThreadSidePanelContent.OpenComposerObserver) never implemented the documented MeshNodeStreamCache.IsTransientOwnerFailure retry contract that every layout area implements via RetryAreaWithBackoff(AreaErrorClassifier.ShouldRetryArea) — the very first transient owner miss surfaced verbatim (this issue's "No response received in hub cache … SubscribeRequest" banner, which AreaErrorClassifierTest already pins as transient). That is exactly what open PR #1177 fixes: both composer readers get the same bounded, backoff-throttled retry as every layout area; a miss that survives the budget logs a Warning instead of popping a raw framework banner.

   Recommend: close via PR #1177 (adding Fixes #1175 to it fits — the composer-reader retry gap is this issue's actionable defect; the underlying activation starvation is #1172's, and the "hub-cache resolution … does it gracefully handle a missing hub or hang until timeout" question in Where to look is answered: it forwards the failure for the caller to retry by contract, and the composer readers were the callers that didn't).
2. rbuergi commented on Aug 10, 2026

   

   rbuergi

   on Aug 10, 2026

   Contributor

   More actions

   Closing — fixed by PR #1177 (merged 21:11Z; the closing keyword was missing from its body, hence the manual close).

   Two layers, per the triage on this thread: the client-side defect — ThreadComposer readers missing the documented transient-retry contract on a ShuttingDown verdict — is what #1177 fixes. The server-side starvation that produced the transient in the first place (mid-bake silo saturating its 64 route-dispatch slots behind stalled _Activity compiles) is tracked in #1172 with a fix in flight, and exposure is already reduced by the daily roll cadence (#1173). Evidence on this thread: every occurrence sat inside a roll's join-to-Ready bake window.

   Recurrences on an image containing #1177 + the #1172 fix would be a new fact — reopen then.
3. rbuergi

   closed this as completedon Aug 10, 2026
4. systemorph-com

   reopened this on Sep 17, 2026
5. systemorph-com commented on Sep 17, 2026

   

   systemorph-comBot

   on Sep 17, 2026

   Contributor

   More actions

   Reopened — this fault is happening again.

   Still occurring: 52 more since the last update (56 total).

   • Last seen: 2026-08-11 07:06:33Z
   • Pods: memex-portal-deployment-6bd9c46f8b-z9t6r, memex-portal-deployment-59479dc7d9-kmwvr

   Recent log lines

   2026-08-11 06:54:08Z memex-portal-deployment-59479dc7d9-kmwvr fail: MeshWeaver.Blazor.Portal.Chat.ThreadChatView[0]
         Loading the composer in Area Composer
         System.TimeoutException: No response received in hub cache/gmIIqhF220CKKxqxgY_J7Q within 00:01:00 for request SubscribeRequest (id=KRWTHyhjVkGYow3rGGGHvg) → target carson/_Thread/ThreadComposer. The request may have been undeliverable or the target hub was not found.
   2026-08-11 07:06:33Z memex-portal-deployment-59479dc7d9-kmwvr fail: MeshWeaver.Blazor.Portal.Chat.ThreadChatView[0]
         Loading the composer in Area Thread/1
         System.TimeoutException: No response received in hub cache/gmIIqhF220CKKxqxgY_J7Q within 00:01:00 for request SubscribeRequest (id=riyrR4b7xEmoL1EtU4HIvQ) → target sglauser/_Thread/is-there-an-about-section-somewhere-on-m-3d37. The request may have been undeliverable or the target hub was not found.
   2026-08-11 07:06:33Z memex-portal-deployment-59479dc7d9-kmwvr fail: MeshWeaver.Blazor.Portal.Chat.ThreadChatView[0]
         Loading the composer in Area Thread/1
         System.TimeoutException: No response received in hub cache/gmIIqhF220CKKxqxgY_J7Q within 00:01:00 for request SubscribeRequest (id=riyrR4b7xEmoL1EtU4HIvQ) → target sglauser/_Thread/is-there-an-about-section-somewhere-on-m-3d37. The request may have been undeliverable or the target hub was not found.
   
6. rbuergi commented on Sep 18, 2026

   

   rbuergi

   on Sep 18, 2026

   Contributor

   More actions

   Closing again — the fix this issue was closed on is still in the code today, and the two post-close samples came from a pod running the pre-fix image. Silent for 38 days.

   Quiet. The reopen of 2026-09-17T07:52:35Z cites Last seen: 2026-08-11 07:06:33Z — 38.0 days before today (2026-09-18). Ingest is live (the incident store was updating at 2026-09-18T05:42Z), so the silence is real. Still occurring: 52 more compares accumulated totals since the last update; it is not a statement about now.

   The client-side defect is fixed and the fix is still present. This was closed on 2026-08-10T21:14:36Z on PR #1177 ("Composer load no longer surfaces a transient owner miss as a raw error", merged 2026-08-10T21:11:42Z). The composer views moved to MeshWeaver.Plugins with the AI/chat surface (#2276); checked there at origin/main efb2449dc3:

   $ git grep -n "RetryAreaWithBackoff\|IsTransientOwnerFailure" origin/main -- '*ThreadChatView*'
   src/MeshWeaver.Blazor.Chat/ThreadChatView.razor.cs:892:  /// retry (<c>MeshNodeStreamCache.IsTransientOwnerFailure</c>) — the bounded backoff here is that
   src/MeshWeaver.Blazor.Chat/ThreadChatView.razor.cs:906:      .RetryAreaWithBackoff(AreaErrorClassifier.ShouldRetryArea)
   

   OpenComposerProjection implements the documented transient-owner-failure retry contract, and its remarks name this exact banner ("No response received in hub … target hub was not found") as the transient it now rides out instead of surfacing.

   The server-side cause is also resolved. The starvation that produced the transient in the first place — mid-bake silo saturating its 64 route-dispatch slots behind stalled _Activity/compile dispatches — was #1172, now closed as completed; exposure was additionally cut by the roll-cadence change in #1173.

   Why the post-close occurrences are not a regression. They land 2026-08-11 06:54:08Z and 07:06:33Z, both on pod memex-portal-deployment-59479dc7d9-kmwvr. This issue's own triage on 2026-08-10 pins that pod to the 13:38:56Z roll of 2026-08-10, joining Orleans at ~13:39 and Ready at 15:59:37 — i.e. an image built hours before #1177 merged at 21:11Z. The recurrence is the pre-fix image still emitting, and it stopped there: nothing in 38 days.

   If this fault recurs, the recurrence bot reopens this issue with fresh evidence — that is the designed behaviour, and it is why closing a dormant incident is safe.
7. rbuergi

   closed this as not plannedon Sep 18, 2026