Thread grain permanently wedged: one async task Running for >1h on the ActivationTaskScheduler, 1376 messages queued, no recovery path short of pod restart #147

Symptom

Threads stop loading on the portal. After several retries the UI shows:

Area unavailable. The view at ‹path› did not become addressable after 5 retries — it may still be initialising or its NodeType may be compiling. Reload to retry.

The portal log contains a delivery timeout for the specific thread grain with the following diagnostic block (condensed):

Delivery to 'AgenticPension/Dashboard/_Thread/change-the-pie-chart-to-a-bar-chart-unde-4367' failed:
Response did not arrive on time in 00:00:30 …
Last known status:
  IsExecuting: False, IsWaiting: True
  NonReentrancyQueueSize: 1376
  NumRunning: 0
  IdlenessTimeSpan: 01:07:53.5440000
  CollectionAgeLimit: 00:15:00
TaskScheduler status:
  WorkGroupStatus=Running
  QueuedWorkItems=1; Total Enqueued=659; Total processed=657
  Executing Task Id=1 Status=Running for 01:07:53.5440000

The thread's durable DB state is already terminal (IsExecuting=false, pendingUserMessages={}) — the round completed. But the grain remains permanently wedged in memory. Recovery requires a pod restart.

Root cause chain

1. Hub ActionBlock runs on the grain's ActivationTaskScheduler

MessageHubGrain.CompleteActivation wires the hosted hub with .WithTaskScheduler(grainScheduler) (line 247). This means all hub message processing — handlers, subscriptions, workspace updates, watchdog actions — runs on the Orleans grain's ActivationTaskScheduler.

2. One async task captures the scheduler and never completes

ExecuteMessageAsync (or its surrounding orchestration) places an async continuation onto the grain's scheduler — either via InvokeAsync(async () => { ... }) or await Task.Run(...) called from inside a turn. The async state machine becomes Task Id=1, Status=Running on the scheduler.

The actual async work (LLM streaming via the IoPool) ran on a thread-pool thread. But the continuation awaiting its result was posted back to the grain's ActivationTaskScheduler. That scheduler has WorkGroupStatus=Running and QueuedWorkItems=1 — the continuation is queued but cannot execute because Task Id=1 is still logically "Running" from the scheduler's perspective, blocking the single scheduler slot.

3. The hub's ActionBlock is therefore also blocked

Because the hub's ActionBlock uses the same grain scheduler (WithTaskScheduler(grainScheduler)), no hub messages can be processed while the scheduler slot is occupied. Every incoming IMessageHubGrain.DeliverMessage call from Orleans piles up in the NonReentrancyQueueSize — which reached 1,376 messages over 1h7m.

4. The stuck-round watchdog cannot rescue the grain

InstallLifecycle (ThreadExecution.cs line 534) installs a watchdog: if the thread node emits nothing for StuckGracePeriod = 90s while IsExecuting=true, it calls workspace.GetMeshNodeStream().Update(...) to force the thread to Idle. This watchdog fires correctly — but its Update call posts a PatchDataRequest to the same thread hub grain whose scheduler is stuck. That message joins the 1,376-message backlog and is never processed.

5. _activeOperations > 0 prevents grain idle-collection

BeginAsyncOperation() (line 1634) calls BeginLongRunningOperation(), which increments _activeOperations and calls DelayDeactivation(10min). A RegisterGrainTimer fires every minute: while _activeOperations > 0, it calls DelayDeactivation(10min) again. If the round's LongRunningOperationScope is never disposed (because the streaming continuation never resumes), _activeOperations stays positive — and DelayDeactivation is called indefinitely, preventing Orleans idle-collection even though CollectionAgeLimit=00:15:00.

Orleans also skips idle-collection when NonReentrancyQueueSize > 0. Both conditions keep the grain alive permanently.

6. No external deactivation path exists

The only way to break the loop today is a pod restart, which disposes the grain and triggers executionCts.Cancel() via RegisterForDisposal, which cancels the AI API call. There is no admin API or MCP tool to force-deactivate a specific grain.

Proposed fixes

Fix 1 — Cap the total DelayDeactivation budget (highest priority)

Add a per-round wall-clock deadline on BeginLongRunningOperation. If _activeOperations > 0 for longer than MaxStreamingDuration (e.g., 30 minutes), stop calling DelayDeactivation and allow natural idle-collection to proceed. When the grain deactivates, executionCts.Cancel() fires via RegisterForDisposal, cancelling the AI API call and breaking the circular dependency.

// In BeginLongRunningOperation or the keep-alive timer:
if (_operationStartedAt.HasValue
    && DateTime.UtcNow - _operationStartedAt.Value > MaxStreamingDuration)
{
    // Stop extending the grain lifetime — let idle-collection deactivate it.
    // executionCts.Cancel() will fire on deactivation and cancel the API call.
    return;
}
DelayDeactivation(TimeSpan.FromMinutes(10));

Fix 2 — Stuck-round watchdog calls RequestDeactivation instead of (or alongside) stream.Update

The watchdog's stream.Update routes through the stuck grain's queue and is never processed. The watchdog should ALSO (or instead) call the Orleans DeactivateOnIdle() API via a grain callback, forcing the grain to deactivate and the CancellationToken to fire:

// In InstallLifecycle, expose the grain's DeactivateOnIdle via a callback:
.Set(new GrainDeactivateCallback(() => DeactivateOnIdle()))

// In the stuck-round watchdog:
hub.Get<GrainDeactivateCallback>()?.Invoke();   // triggers executionCts.Cancel via RegisterForDisposal

This deactivates the grain immediately without going through the hub's ActionBlock, bypassing the stuck scheduler.

Fix 3 — Add an admin endpoint or MCP tool to force-deactivate a grain by path

Expose IManagementGrain.DeactivateActivation(grainId) (Orleans built-in) through a portal admin endpoint or the MeshWeaver MCP server's execute_script surface. This would allow recovery without a pod restart.

Fix 4 — Add an independent streaming timeout (defence in depth)

The AI streaming call (IIoPool.Invoke(async ct => { await foreach ... })) should carry its own wall-clock CancellationTokenSource linked to executionCts with a hard upper limit (e.g., CancellationTokenSource.CreateLinkedTokenSource(executionCts.Token, new CancellationTokenSource(MaxStreamingDuration).Token)). This ensures the streaming call itself times out regardless of the grain lifecycle.

Files to edit

Affected deployment

Confirmed on client-i.meshweaver.cloud (thread AgenticPension/Dashboard/_Thread/change-the-pie-chart-to-a-bar-chart-unde-4367, wedge duration ≥ 1h7m, NonReentrancyQueueSize=1376). The pattern is general: any distributed portal where an LLM streaming call hangs without a timeout is exposed to this permanent wedge.