Fix hosted/recorded sessions getting stuck "active" (orphaned watcher + daemon session-end)

docs/superpowers/specs/2026-06-13-hosted-session-server-reconcile-spec.md

@@ -0,0 +1,99 @@
+# Server-side reconciliation of stuck hosted/recorded sessions
+
+**Date:** 2026-06-13
+**Status:** Spec (for the kcap-**server** repo — cannot be implemented from kcap-cli)
+**Related:** client-side fixes in this repo (watcher watchdog, daemon `EndAgentSession`); design `2026-06-13-permission-bridge-reconnect-retry-design.md`
+
+## Problem (observed incident)
+
+Session `9687912eed69420d9a21ba1c66b7698d` (a daemon-hosted agent) showed as **active
+forever** on the server after the hosted `claude` process had already died. Forensics:
+
+- The hosted `claude` PID was dead.
+- Its two `kcap watch` sidecar processes were **orphaned but still alive**, holding live
+  SignalR connections to the server (`…:443 ESTABLISHED`), 1h+ after the agent died.
+- The hosting daemon had been replaced by a new daemon instance that had no knowledge of
+  the session, so it never sent `EndAgentSession`.
+
+So "active" was being held open by (a) orphaned watcher connections that never sent
+session-end, and (b) a missing `EndAgentSession` for the agent-run.
+
+## Why the client side alone can't guarantee this
+
+There are three independent producers of "this session is active" and three matching
+failure modes. The client-side fixes (this repo) close two of them, but a daemon/agent
+that dies **hard** (SIGKILL, crash, host reboot, daemon replacement) can leave residue
+the client can never clean up — by definition the cleanup code is dead. The server is
+the only component guaranteed to be alive to reconcile. Defense in depth requires a
+server-side backstop.
+
+Client-side fixes already in flight (for reference, not server work):
+
+1. **Watcher watchdog** — `kcap watch` now reliably resolves the durable coding-agent PID
+   and self-terminates + POSTs `session-end` when the agent dies (previously the watchdog
+   silently never started for Claude, orphaning the watcher).
+2. **Daemon `EndAgentSession`** — daemon attempts to end the hosted AgentSession on stop /
+   agent-exit / shutdown.
+
+Neither survives a hard daemon/host death. Hence this spec.
+
+## Goal
+
+The server must converge a session/agent-run to a terminal state when its producers are
+demonstrably gone, without waiting on a client message that may never arrive.
+
+## Proposed reconciliation (server)
+
+Two triggers, each with a short grace period to tolerate transient reconnects (the
+cloudflared-instability pattern — connections drop and re-establish within seconds):
+
+### A. Daemon disconnect → end that daemon's in-flight agent-runs
+
+When a daemon's hub connection drops and is **not** re-established (by the same daemon
+identity / `(owner, name)` slot) within a grace window (suggest **60–90s**, comfortably
+above the daemon's auto-reconnect cadence which retries at ≤30s):
+
+- Mark every `AgentRun`/`AgentSession` that daemon owned and that is still `active` as
+  ended, reason `daemon_disconnected` (distinct from `agent_stopped`/`agent_exited` so it's
+  attributable in the read model).
+- This is the backstop the daemon code already *assumes* exists — see the comments in
+  `AgentOrchestrator.cs` (`ReadAgentOutputAsync` finally, `CleanupAgentAsync`,
+  `DisposeAsync`) which skip `EndAgentSession` on shutdown "because the server detects the
+  daemon disconnection and ends its sessions on its own." **This assumption must be
+  verified; the incident suggests it is not happening (or not for replaced daemons).**
+
+### B. Watcher disconnect → end sessions with no remaining live producer
+
+When the last `kcap watch` connection for a session drops and is not re-established within
+a grace window (suggest **60–90s**), and there is no other live producer (no connected
+daemon hosting it, no active local-CLI watcher):
+
+- Mark the session ended, reason `watcher_disconnected`.
+- Optionally trigger the same post-session work the normal `session-end` path does
+  (what's-done generation) if not already done — idempotently.
+
+### Idempotency
+
+Both triggers must be **idempotent** with the existing `SessionEnded` / `EndAgentSession`
+paths: if the client later reconnects and sends session-end (or the daemon's
+`EndAgentSession` lands), it must be a no-op, not a duplicate end or a resurrection.
+
+## Open questions for the server team
+
+1. **Does the server already react to daemon hub disconnect at all today?** The daemon code
+   asserts it does. If yes: why did the replaced-daemon case (incident) leave the session
+   active — is reconciliation keyed on connection-id rather than daemon identity, so a
+   *replacement* daemon connecting masks the old one's disconnect without ending its runs?
+2. **Is "active session" derived from a live watcher connection, from `SessionEnded`
+   absence, or both?** Determines whether trigger B is needed or whether watcher-connection
+   teardown already clears it.
+3. **Grace-period source of truth** — align with the daemon heartbeat (7s tick / 5s ping
+   deadline) and auto-reconnect (≤30s) so the window can't fire during a normal blip.
+
+## Acceptance
+
+- Kill a hosted agent's `claude` and its watchers without any client-side session-end →
+  session converges to ended within the grace window, attributed `daemon_disconnected` /
+  `watcher_disconnected`.
+- A transient reconnect within the grace window does **not** end the session.
+- Re-delivery of a client session-end afterwards is a no-op.

src/Capacitor.Cli.Daemon/Services/AgentOrchestrator.cs

@@ -91,6 +91,14 @@ internal partial class AgentOrchestrator : IAsyncDisposable {
 
     static readonly TimeSpan PingDeadline = TimeSpan.FromSeconds(5);
 
+    /// <summary>
+    /// How long <see cref="FinalizeAgentRunAsync"/> waits on the (reconnect-retrying)
+    /// EndAgentSession call before proceeding with local cleanup regardless. Covers a
+    /// typical transient SignalR blip with margin; a longer outage proceeds to cleanup
+    /// while the retry continues in the background. Settable so tests don't wait 30s.
+    /// </summary>
+    internal TimeSpan EndAgentSessionBudget { get; set; } = TimeSpan.FromSeconds(30);
+
     // Linked to IHostApplicationLifetime.ApplicationStopping so the shutdown gate
     // trips as soon as the host begins stopping — the same instant ServerConnection's
     // _ct (also ApplicationStopping) cancels SignalR calls. Otherwise there's a
@@ -526,27 +534,41 @@ async Task FinalizeAgentRunAsync(AgentInstance agent) {
             // its own session-end hook on SIGTERM/exit, so without this call the
             // session would stay "active" forever in the read model. Server-side is
             // idempotent — if claude did fire session-end first, this is a no-op.
-            // Reason is read from agent.PendingEndReason so that if HandleStopAgent's
-            // own call failed (transient SignalR error) and this finally-block call
-            // is the one that lands, a user-initiated stop is still recorded as
-            // "agent_stopped" rather than "agent_exited".
+            // Reason is read from agent.PendingEndReason so a user-initiated stop is
+            // recorded as "agent_stopped" rather than "agent_exited".
+            //
+            // EndAgentSessionAsync retries across SignalR reconnects, so it can block
+            // for the length of an outage. We must NOT let that stall local cleanup
+            // (worktree/process disposal, removing the agent from _agents), so bound how
+            // long we WAIT on it to EndAgentSessionBudget. The retry keeps running in the
+            // background — a reconnect shortly after still lands the session-end — and a
+            // genuinely long outage falls back to server-side daemon-disconnect reconcile.
+            var endTask = _server.EndAgentSessionAsync(agent.Id, agent.PendingEndReason);
+
             try {
-                var result = await _server.EndAgentSessionAsync(agent.Id, agent.PendingEndReason);
+                var result = await endTask.WaitAsync(EndAgentSessionBudget, _shutdownCts.Token);
 
                 // The daemon doesn't track sessionId on its own (only agentId), so
                 // the server returns it in the result. Spawn what's-done locally
                 // when the server says yes.
                 if (result is { GenerateWhatsDone: true, SessionId: not null }) {
                     SpawnWhatsDoneGenerator(result.SessionId);
                 }
+            } catch (TimeoutException) {
+                // Outage outlasted the budget. Don't block cleanup; the retry continues
+                // in the background (observed below so a later fault isn't unobserved).
+                LogEndSessionTimedOut(agent.Id, EndAgentSessionBudget.TotalSeconds);
+                ObserveEndSessionInBackground(endTask, agent.Id);
             } catch (OperationCanceledException) when (_shutdownCts.IsCancellationRequested) {
-                // Shutdown fired mid-call — connection is being torn down. Server
+                // Shutdown fired mid-wait — connection is being torn down. Server
                 // detects daemon disconnection independently. No warning needed.
             } catch (Exception ex) {
                 LogEndSessionFailed(ex, agent.Id);
             }
 
-            // Clean up worktree and unregister from server
+            // Clean up worktree and unregister from server. Runs unconditionally — even
+            // when end-session timed out and is still retrying in the background — so a
+            // prolonged outage can never pin the agent in _agents or leak its worktree.
             await CleanupAgentAsync(agent.Id);
         } catch (Exception ex) {
             LogCleanupError(ex, agent.Id);
@@ -605,29 +627,36 @@ async Task HandleStopAgent(string agentId) {
                 LogGracefulExitTimedOut(agentId, GracefulExitWait.TotalSeconds);
             }
 
-            // Tell the server to end the session. Idempotent server-side: if claude
-            // did fire session-end during the graceful window, this is a no-op
-            // (returns GenerateWhatsDone=false and the CLI's session-end handler
-            // already spawned the what's-done generator on its end).
-            try {
-                var result = await _server.EndAgentSessionAsync(agentId, agent.PendingEndReason);
-
-                if (result is { GenerateWhatsDone: true, SessionId: not null }) {
-                    SpawnWhatsDoneGenerator(result.SessionId);
-                }
-            } catch (Exception ex) {
-                LogEndSessionFailed(ex, agentId);
-            }
-
-            // Cancel the read loop, then terminate the process if it didn't exit gracefully.
-            // The read loop's finally block will handle CleanupAgentAsync.
+            // Cancel the read loop and terminate the process. We deliberately do NOT end
+            // the AgentSession here: EndAgentSessionAsync now retries across SignalR
+            // reconnects (so it can block while a dropped connection recovers), and a
+            // user-initiated stop must not wait on that. Cancelling ReadCts unblocks the
+            // read loop, whose finally block runs FinalizeAgentRunAsync once the process
+            // exits — that ends the session (with retry) using agent.PendingEndReason
+            // ("agent_stopped") and spawns the what's-done generator if the server asks.
+            // So session-end is reliable as the post-exit backstop without delaying
+            // teardown, and is idempotent: if claude already fired its own session-end
+            // during the graceful window above, the backstop call is a server-side no-op.
             await agent.ReadCts.CancelAsync();
             await agent.Process.TerminateAsync(TimeSpan.FromSeconds(10));
         } catch (Exception ex) {
             LogStopError(ex, agentId);
         }
     }
 
+    /// <summary>
+    /// Observes a background EndAgentSession retry that outlived the finalize budget so a
+    /// later fault isn't an unobserved task exception. Success and shutdown-cancellation
+    /// are intentionally ignored; only a genuine fault is logged.
+    /// </summary>
+    void ObserveEndSessionInBackground(Task<EndAgentSessionResult> endTask, string agentId) =>
+        _ = endTask.ContinueWith(
+            t => LogEndSessionFailed(t.Exception!.GetBaseException(), agentId),
+            CancellationToken.None,
+            TaskContinuationOptions.OnlyOnFaulted | TaskContinuationOptions.ExecuteSynchronously,
+            TaskScheduler.Default
+        );
+
     /// <summary>
     /// Spawns <c>kcap generate-whats-done {sessionId}</c> as a detached process.
     /// Used when the daemon-driven session-end path supplants claude's own session-end
@@ -1018,6 +1047,9 @@ static string ExtractTerminalText(TerminalOutputBuffer buffer) {
     [LoggerMessage(Level = LogLevel.Warning, Message = "Failed to end session for agent {AgentId} (server may not record SessionEnded)")]
     partial void LogEndSessionFailed(Exception ex, string agentId);
 
+    [LoggerMessage(Level = LogLevel.Warning, Message = "EndAgentSession for agent {AgentId} did not complete within {Seconds}s; proceeding with cleanup while the retry continues in the background (server reconciles on daemon disconnect)")]
+    partial void LogEndSessionTimedOut(string agentId, double seconds);
+
     [LoggerMessage(Level = LogLevel.Information, Message = "Spawned what's-done generator for session {SessionId} (PID {Pid})")]
     partial void LogWhatsDoneSpawned(string sessionId, int pid);
 

src/Capacitor.Cli.Daemon/Services/ServerConnection.cs

@@ -20,6 +20,7 @@ internal partial class ServerConnection : IAsyncDisposable, IDaemonHeartbeatPort
     readonly RegistrationGate          _gate = new();
 
     static readonly TimeSpan PermissionRetryPollInterval = TimeSpan.FromMilliseconds(500);
+    static readonly TimeSpan EndSessionRetryPollInterval  = TimeSpan.FromMilliseconds(500);
 
     // Events for incoming commands from server
     public event Func<LaunchAgentCommand, Task>?    OnLaunchAgent;
@@ -439,7 +440,13 @@ public virtual Task LaunchFailedAsync(string agentId, string reason)
     /// behaviour of the CLI session-end handler for the local-claude case.
     /// </summary>
     public virtual Task<EndAgentSessionResult> EndAgentSessionAsync(string agentId, string reason)
-        => _hub.InvokeAsync<EndAgentSessionResult>("EndAgentSession", agentId, reason, cancellationToken: _ct);
+        => ConnectionRetry.InvokeWithConnectionRetryAsync(
+            () => _hub.InvokeAsync<EndAgentSessionResult>("EndAgentSession", agentId, reason, cancellationToken: _ct),
+            () => IsReady,
+            EndSessionRetryPollInterval,
+            attempt => LogEndSessionRetry(agentId, attempt),
+            _ct
+        );
 
     /// <summary>
     /// Forwards a hosted-agent permission request to the server's <c>RequestPermission</c>
@@ -646,6 +653,9 @@ public async ValueTask DisposeAsync() {
     [LoggerMessage(Level = LogLevel.Information, Message = "RequestPermission for session {SessionId} interrupted by a connection drop (retry {Attempt}); waiting for the daemon connection to recover before retrying")]
     partial void LogPermissionRetry(string sessionId, int attempt);
 
+    [LoggerMessage(Level = LogLevel.Information, Message = "EndAgentSession for agent {AgentId} interrupted by a connection drop (retry {Attempt}); waiting for the daemon connection to recover before retrying")]
+    partial void LogEndSessionRetry(string agentId, int attempt);
+
     [LoggerMessage(Level = LogLevel.Warning, Message = "Failed to post agent run event, retrying in {Delay}s")]
     partial void LogEventPostFailed(Exception ex, double delay);
 

src/Capacitor.Cli/Commands/WatchCommand.cs

@@ -11,6 +11,30 @@
 namespace Capacitor.Cli.Commands;
 
 static partial class WatchCommand {
+    /// <summary>Outcome of deciding whether the parent-exit watchdog can run.</summary>
+    internal enum ParentWatchdog {
+        /// <summary>Parent PID is alive — start the 5s liveness poll.</summary>
+        Monitor,
+
+        /// <summary>No parent PID was supplied — nothing to monitor.</summary>
+        NoParentPid,
+
+        /// <summary>A parent PID was supplied but it's already dead at startup
+        /// (typically a transient process from bad PID resolution). Must be surfaced,
+        /// never silently skipped.</summary>
+        ParentAlreadyDead
+    }
+
+    /// <summary>
+    /// Decides whether the parent-exit watchdog should run. Pure so the three
+    /// outcomes — including the dead-at-startup case that caused stuck sessions —
+    /// are unit-testable without spawning processes.
+    /// </summary>
+    internal static ParentWatchdog DecideParentWatchdog(int? parentPid, Func<int, bool> isAlive) =>
+        parentPid is not { } ppid ? ParentWatchdog.NoParentPid
+        : !isAlive(ppid)          ? ParentWatchdog.ParentAlreadyDead
+        :                           ParentWatchdog.Monitor;
+
     public static async Task<int> RunWatch(
             string  baseUrl,
             string  sessionId,
@@ -85,26 +109,47 @@ public static async Task<int> RunWatch(
             ctx.Cancel = true;
         });
 
-        // Watch the spawning claude process. If it dies without firing session-end
-        // (crash, force-kill, IDE-detach), self-terminate within ~5s instead of orphaning.
-        if (parentPid is { } ppid && ProcessHelpers.IsProcessAlive(ppid)) {
-            Log($"Monitoring parent pid {ppid}");
-            _ = Task.Run(async () => {
-                while (!cts.Token.IsCancellationRequested) {
-                    try {
-                        await Task.Delay(TimeSpan.FromSeconds(5), cts.Token);
-                    } catch (OperationCanceledException) {
-                        return;
-                    }
+        // Watch the spawning coding-agent process. If it dies without firing
+        // session-end (crash, force-kill, IDE-detach), self-terminate within ~5s and
+        // POST session-end instead of orphaning. Crucially, the cases where we DON'T
+        // monitor are now logged: a silently-skipped watchdog is exactly the failure
+        // that left sessions stuck "active" with the watcher still connected — the
+        // resolved parent PID was already dead at startup and nothing recorded it.
+        switch (DecideParentWatchdog(parentPid, ProcessHelpers.IsProcessAlive)) {
+            case ParentWatchdog.NoParentPid:
+                Log("No parent pid supplied; parent-exit watchdog disabled (session-end relies on the agent's own hook)");
 
-                    if (!ProcessHelpers.IsProcessAlive(ppid)) {
-                        Log($"Parent pid {ppid} exited; shutting down watcher");
-                        Interlocked.Exchange(ref parentExited, 1);
-                        cts.Cancel();
-                        return;
+                break;
+
+            case ParentWatchdog.ParentAlreadyDead:
+                Log($"Parent pid {parentPid} already dead at watcher startup; parent-exit watchdog NOT started — "
+                  + "session-end will not be POSTed if the agent ends abruptly. This usually means parent-PID "
+                  + "resolution returned a transient process; see ProcessHelpers.GetCodingAgentPid.");
+
+                break;
+
+            case ParentWatchdog.Monitor:
+                var ppid = parentPid!.Value;
+                Log($"Monitoring parent pid {ppid}");
+                _ = Task.Run(async () => {
+                    while (!cts.Token.IsCancellationRequested) {
+                        try {
+                            await Task.Delay(TimeSpan.FromSeconds(5), cts.Token);
+                        } catch (OperationCanceledException) {
+                            return;
+                        }
+
+                        if (!ProcessHelpers.IsProcessAlive(ppid)) {
+                            Log($"Parent pid {ppid} exited; shutting down watcher");
+                            Interlocked.Exchange(ref parentExited, 1);
+                            cts.Cancel();
+
+                            return;
+                        }
                     }
-                }
-            }, cts.Token);
+                }, cts.Token);
+
+                break;
         }
 
         var state = new WatchState();