Balanced-mode host shutdown hangs indefinitely when the node holds a large number of agents

[jeremydmiller]

IHost.StopAsync() never returns when a Balanced-mode Wolverine node is shut down while holding a large agent load. Found while trying to wire SlowTests into CI for #3779 — this is what blocks it.

Reproduction

docker compose up -d postgresql
dotnet test src/Testing/SlowTests/SlowTests.csproj --framework net9.0 \
  --filter 'FullyQualifiedName~agent_reassignment_at_scale'

The test's assertions all pass. DisposeAsync then stops the three hosts in reverse creation order; the first two stop cleanly and the third never returns. dotnet test therefore never reports, and the process has to be killed. Reproduced 4/4, including on a completely idle machine.

The controlled pair

The only change is a bound on the teardown stop in agent_reassignment_at_scale.DisposeAsync:

DisposeAsync	Result
await host.StopAsync() (as committed)	hangs indefinitely; run never reports
await host.StopAsync().WaitAsync(20.Seconds())	passed, 2m18s

So the hang is entirely in host shutdown. Nothing about the test's own logic is involved — with the stop bounded, the suite is green.

What is known about the wedge

• It is the leader that hangs. _hosts.Reverse() means the last host stopped is the first one created — node 1, the leader, the node that was running the whole old generation. The other two shut down normally: the assignment table drains 478 → 322 → 192 and then stops moving.
• It is not the simulated slow stop. DisposeAsync sets _telemetry.StopDelay = TimeSpan.Zero before stopping anything, so the fake agents stop instantly during teardown.
• Nothing is running. dotnet-stack report against the wedged process shows every managed thread parked — thread-pool workers idle on LowLevelLifoSemaphore.Wait, the main thread blocked in the xUnit entry point, no CPU burn at all (11s of CPU across 40 minutes of wall clock). That rules out a spin or a lock convoy and points at an await that is never completed.
• The runtime goes quiet mid-shutdown. The node's wolverine_nodes.health_check row stops updating roughly two minutes in, and the row is never deleted — so shutdown gets far enough to stop the health-check loop and never far enough to deregister the node.

Scale that provokes it

agent_reassignment_at_scale runs 3 nodes over a 480-agent universe with the shipped defaults (AgentStartBatchSize = 50, MaxAgentStartParallelism = 10), 240 agents starting on node 1 before two more nodes join. The sibling tests in the same project — agent_assignment_at_scale and slow_starts_outrun_reply_windows — complete normally (3m51s for the pair), so this is not every Balanced-mode shutdown; something about a leader shutting down after a large reassignment wave is the trigger.

Why it matters beyond the test

A host that never completes StopAsync is a deployment that never completes a rolling restart: the orchestrator waits out its termination grace period and then SIGKILLs the pod, which for a durable node means the shutdown path that deregisters the node and releases its agents never runs. That is the same starting condition as the GH-3753 chain.

Blocks

• Simulate slow agent starts in the distribution harness so #3753's convergence failures reproduce at dev scale #3779 — SlowTests cannot be added to a CI job while one of its tests wedges the runner. Bounding the teardown inside the test would make CI green while hiding this.

🤖 Generated with Claude Code

https://claude.ai/code/session_0116vfBcKwcjWn8msM4ZjkuA

[jeremydmiller]

Root-caused, from a dump of the wedged process

dotnet-stack report is the wrong tool here — an await that never completes lives on the GC heap,
not on a thread stack, which is why it showed nothing but parked thread-pool workers. dotnet-dump collect + dumpasync --coalesce gives the whole chain verbatim:

Task<AgentPresenceReport>                                    <-- pending
 MessageRoute.sendAndAwaitReplyAsync<AgentPresenceReport>
  MessageRoute.RemoteInvokeAsync<AgentPresenceReport>
   WolverineRuntime.InvokeAsync<AgentPresenceReport>
    AgentWorkConfirmation.<AwaitAsync>b__0            (the QueryAgentPresence poll)
     AgentWorkConfirmation.AwaitCoreAsync
      AssignAgents.ExecuteAsync
       AgentCommandHandler.HandleAsync
        Executor.InvokeAsync / InvokeInlineAsync
         WolverineRuntime.<startNodeAgentWorkflowAsync>b__39_0   (the dispatcher's executor)
          AgentCommandDispatcher.runLaneAsync
           AgentCommandDispatcher.DisposeAsync        <-- awaiting the lane worker
            WolverineRuntime.teardownAgentsAsync
             WolverineRuntime.StopAsync
              Host.ForeachService<IHostedService> / Host.StopAsync
               agent_reassignment_at_scale.DisposeAsync

A second stack holds the reply that confirmation is racing: AssignAgents.invokeStartsAsync ->
InvokeAsync<AgentsStarted> -> Task<AgentsStarted>, also pending.

So the leader is stuck in AgentCommandDispatcher.DisposeAsync(), waiting on a lane worker that is
executing an AssignAgents chunk aimed at a node that has already shut down.

Correction to the report: it is not indefinite

I let the wedged run go to completion rather than killing it:

Passed! - Failed: 0, Passed: 1, Skipped: 0, Total: 1, Duration: 17 m 17 s - SlowTests.dll (net9.0)

Against the control in this issue (.WaitAsync(20.Seconds()) -> 2m18s) that is ~15 minutes
spent inside Host.StopAsync(), after which it released and the test reported green. 15 minutes is
AgentBatchTimeouts.ReplyWindowFor(30) = 30s + 30 x 30s = 930s — exactly one reply window for the
chunk in flight.

The accurate statement is one full agent-batch reply window per queued command, per lane —
25.5 minutes at the shipped AgentStartBatchSize = 50 — not "never returns". The 40-minute
observation is consistent with a deeper lane queue; nobody had waited it out. That distinction
matters for the fix: this is not a deadlock to break, it is a timeout being paid in full during
shutdown because the cancellation that should short-circuit it is dropped on the floor.

Three compounding defects

1. The dispatcher holds the wrong cancellation token

WolverineRuntime.Agents.cs:313-314

_dispatcher = new AgentCommandDispatcher(
    async (command, token) => await new MessageBus(this).InvokeAsync<AgentCommands>(command, token),
    Logger, Cancellation);          // <-- runtime-wide Cancellation

Every other agent-side component takes _agentCancellation.Token — NodeAgentController (:275),
DeferredAgentCommandRunner (:283). The dispatcher is the odd one out, and the ordering in
WolverineRuntime.HostService.cs:350-435 makes that fatal:

step	line	effect
await _agentCancellation.CancelAsync()	363	cancels the agent token
endpoints drain / stores drain / release ownership	377-420
await teardownAgentsAsync()	425	awaits _dispatcher.DisposeAsync()
DurabilitySettings.Cancel()	428	cancels Cancellation — too late

The token that would end the lane loop (while (!_cancellation.IsCancellationRequested)) and make
the in-flight command bail out of AwaitCoreAsync is only cancelled after the thing waiting on it
has already returned.

2. DisposeAsync drains the queue instead of abandoning it

It calls lane.Queue.Writer.TryComplete() and then awaits each lane's worker. Completing a channel
writer does not discard buffered items — ReadAsync keeps handing them out. So shutdown pays
for the in-flight command and everything still queued behind it, serially per lane.

3. ReplyListener silently drops an already-cancelled token

ReplyListener.cs:13-27

public ReplyListener(Envelope envelope, ReplyTracker parent, TimeSpan timeout,
    CancellationToken cancellationToken, ResultTypeRegistry? resultTypes = null)
{
    ...
    cancellationToken.Register(onCancellation);      // <-- _completion is still null here

    _completion = new TaskCompletionSource<T>(TaskCreationOptions.RunContinuationsAsynchronously);
    _cancellation = new CancellationTokenSource(timeout);
    _cancellation.Token.Register(onCancellation);

onCancellation ends in _completion?.TrySetException(...). When cancellationToken is already
cancelled, Register invokes the callback synchronously, at which point _completion is still
null — the null-conditional swallows the completion. The listener then waits its full timeout
instead of failing fast.

That is why _agentCancellation already being cancelled buys nothing: WolverineRuntime.InvokeAsync<T>
passes _agentCancellation.Token into every agent request, and during shutdown that token is always
already cancelled.

Proved directly against ReplyTracker:

test	result
token cancelled before RegisterListener -> task faults within 2s?	FAIL (waits the full 30s window)
token cancelled after RegisterListener -> task faults within 2s?	PASS

Note this one is not agent-specific: any InvokeAsync<T> in Wolverine handed an already-cancelled
token waits its whole reply window rather than failing fast. Agent shutdown is just where that window
is 25 minutes long.

Bonus: the host's shutdown timeout cannot save it either

WolverineRuntime.StopAsync(CancellationToken cancellationToken) accepts the host's shutdown token
and never uses it — the only occurrence of cancellationToken in the method body is the
parameter declaration. So HostOptions.ShutdownTimeout is inert for this and every other slow
teardown path; the generic host's ForeachService just waits, exactly as the dump shows.

Everything else in the report holds

It is the leader; it is not the simulated slow stop; nothing is running; and the node row is never
deleted because NodeController.StopAsync — which does the DeleteAsync — sits behind
_dispatcher.DisposeAsync() in teardownAgentsAsync.

Fix

1. Build the dispatcher with _agentCancellation.Token, matching every other agent component.
2. Assign _completion before registering the cancellation callback in ReplyListener.
3. Bound AgentCommandDispatcher.DisposeAsync: abandon still-queued commands on shutdown and wait
   the lane workers against a shutdown budget, so no future non-cancellable await can wedge
   IHost.StopAsync() again.

PR to follow.