Fix hosted/recorded sessions getting stuck "active" (orphaned watcher + daemon session-end)

[alexeyzimarev]

Problem

A daemon-hosted Claude session (9687912e…) showed active forever on the server after its hosted claude process had already died. Forensics: the claude PID was dead, but its two kcap watch sidecars were orphaned but still alive, holding live SignalR connections — and the hosting daemon had been replaced without sending EndAgentSession. So "active" was held open by orphaned watcher connections that never POSTed session-end, plus a missing EndAgentSession.

Root cause

The kcap watch parent-exit watchdog (self-terminate + POST session-end when the agent dies) silently never started for Claude. GetCodingAgentPid() used getpgrp(), which resolves the durable agent for Codex but a transient hook process group for Claude — already dead by the time the watcher's IsProcessAlive startup guard ran, so monitoring was skipped with no log line. Quantified across ~/.config/kcap/logs: every recent Claude watcher had 0 Monitoring parent pid lines; only Codex had them.

Changes

1. Watcher PID resolution (CLI) — resolve the agent PID by walking the ppid ancestry and matching the agent by name (claude/codex), robust to each vendor's process-group/launcher topology. Adds per-PID introspection (macOS proc_pidinfo, Linux /proc/<pid>/stat). Hardens the watcher startup guard so a dead-at-startup parent PID is logged loudly instead of silently skipped.
2. Daemon EndAgentSession resilience — routes EndAgentSessionAsync through the existing ConnectionRetry helper (same one RequestPermission uses) so it survives a transient SignalR drop. HandleStopAgent now terminates the process first and lets the post-exit FinalizeAgentRunAsync backstop end the session (with retry), so a stop is never delayed by a reconnecting end-session call.
3. Server-side reconcile spec — documents the server backstop (daemon/watcher disconnect → grace-windowed reconcile) for the kcap-server repo, since a hard daemon/host death can't be cleaned up client-side.

Testing

• 17 new unit tests (TDD): ancestry resolver, GetProcessInfo, vendor-aware GetCodingAgentPid, DecideParentWatchdog, and a daemon test proving a blocked EndAgentSession no longer delays stop termination.
• Full unit suite 1473/1473. CLI + daemon AOT publish clean (no IL2026/IL3050).
• The reconnect-retry helper (ConnectionRetry/RegistrationGate) is already on main (Retry hosted-agent permission requests across SignalR reconnects #152); this builds on it.

Not covered here: integration tests needing a live server; the server-side reconcile (separate repo).

🤖 Generated with Claude Code

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0)

1. End-session retry blocks cleanup ✓ Resolved 🐞 Bug ☼ Reliability

Description

ServerConnection.EndAgentSessionAsync now retries until daemon shutdown, and
AgentOrchestrator.FinalizeAgentRunAsync awaits it before calling CleanupAgentAsync. During a
prolonged SignalR outage this can stall cleanup indefinitely, leaving agents stuck in _agents and
worktrees/process resources uncollected until the daemon exits.

Code

src/Capacitor.Cli.Daemon/Services/ServerConnection.cs[R443-449]

+        => ConnectionRetry.InvokeWithConnectionRetryAsync(
+            () => _hub.InvokeAsync<EndAgentSessionResult>("EndAgentSession", agentId, reason, cancellationToken: _ct),
+            () => IsReady,
+            EndSessionRetryPollInterval,
+            attempt => LogEndSessionRetry(agentId, attempt),
+            _ct
+        );

Evidence

The new EndAgentSession implementation uses ConnectionRetry, whose loop is documented and
implemented as infinite until the passed cancellation token is canceled; the daemon lifetime token
is not canceled during normal outages. FinalizeAgentRunAsync awaits EndAgentSessionAsync and only
then calls CleanupAgentAsync, so an indefinitely-waiting end-session call prevents cleanup from
running.

src/Capacitor.Cli.Daemon/Services/ServerConnection.cs[429-449]
src/Capacitor.Cli.Daemon/Services/ConnectionRetry.cs[21-55]
src/Capacitor.Cli.Daemon/Services/AgentOrchestrator.cs[473-551]
src/Capacitor.Cli.Daemon/Services/AgentOrchestrator.cs[882-905]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`EndAgentSessionAsync` is now wrapped in `ConnectionRetry.InvokeWithConnectionRetryAsync`, whose retry loop is explicitly unbounded (until daemon shutdown). `FinalizeAgentRunAsync` awaits this call before running `CleanupAgentAsync`, so a long-lived disconnect can prevent agent/worktree cleanup for the life of the daemon.

## Issue Context
This is different from the user-initiated stop path (which no longer awaits `EndAgentSessionAsync`), but it still affects the read-loop finalization/backstop path that runs after process exit.

## Fix Focus Areas
- src/Capacitor.Cli.Daemon/Services/ServerConnection.cs[429-449]
- src/Capacitor.Cli.Daemon/Services/ConnectionRetry.cs[21-55]
- src/Capacitor.Cli.Daemon/Services/AgentOrchestrator.cs[473-551]

## Suggested fix approach
1. Keep the retry behavior for transient drops, but **add a bounded time budget** for end-session (e.g., 10–30s) so finalization can proceed.
  - Option A (recommended): in `FinalizeAgentRunAsync`, wrap the awaited call with a timeout CTS (`CreateLinkedTokenSource(_shutdownCts.Token)` + `CancelAfter(...)`) and on timeout log a warning and continue.
  - Option B: run end-session as a background task and always proceed to `CleanupAgentAsync` (still log failures). If you choose this, ensure exceptions are observed/logged.
2. Ensure cleanup (`CleanupAgentAsync`) executes even when end-session fails or times out.
3. Add/adjust a unit test that simulates `IsReady == false` / connection never recovering and asserts cleanup still happens (agent removed from `_agents`, worktree cleanup attempted).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Case-sensitive vendor gate ✓ Resolved 🐞 Bug ≡ Correctness

Description

GetCodingAgentPid(vendor) only enables the new ancestry-walk resolver when vendor is exactly
"claude" or "codex" (case-sensitive), but Program passes --vendor through unnormalized. Casing
variants (e.g. "Claude") will fall back to the legacy PID heuristic and can also cause parent-exit
session-end to be skipped due to WatchCommand’s Ordinal vendor whitelist.

Code

src/Capacitor.Cli/ProcessHelpers.cs[R244-247]

+        if (vendor is "claude" or "codex"
+         && ResolveCodingAgentPid(getppid_native(), vendor, GetProcessInfo) is { } agentPid) {
+            return agentPid;
+        }

Evidence

Program does not normalize the --vendor value before passing it into WatchCommand, while
ProcessHelpers’ new resolver gate only triggers on an exact-case match. Independently,
WatchCommand’s session-end allowlist uses an Ordinal comparer, so casing differences are treated as
unknown and skip the POST.

src/Capacitor.Cli/ProcessHelpers.cs[239-255]
src/Capacitor.Cli/Program.cs[503-538]
src/Capacitor.Cli/Commands/WatchCommand.cs[350-376]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new vendor-aware PID resolver is gated by a case-sensitive vendor match (`vendor is "claude" or "codex"`). `kcap watch` accepts `--vendor` and passes it through without normalization, and `WatchCommand`’s `KnownVendors` also uses `StringComparer.Ordinal`, so user-provided casing can unexpectedly disable both (a) the new PID ancestry resolver and (b) parent-exit session-end POST.

## Issue Context
This is most visible when a user manually runs `kcap watch ... --vendor Claude` (or via scripts). In that case the watcher may regress to the legacy Unix PID heuristic (the Claude failure mode this PR is addressing) and the parent-exit session-end backstop can be skipped.

## Fix Focus Areas
- src/Capacitor.Cli/ProcessHelpers.cs[239-255]
- src/Capacitor.Cli/Program.cs[503-538]
- src/Capacitor.Cli/Commands/WatchCommand.cs[350-376]

## Suggested fix approach
1. Normalize vendor tokens at ingestion (e.g., `watchVendor = (GetArg(...) ?? "claude").Trim().ToLowerInvariant();`).
2. Consider making `KnownVendors` case-insensitive (`StringComparer.OrdinalIgnoreCase`) so the defense-in-depth allowlist remains effective without being brittle.
3. Adjust `GetCodingAgentPid(string? vendor)` to either lower-case the vendor before the comparison or use `Equals(..., OrdinalIgnoreCase)`.
4. Add a small unit test verifying that `--vendor Claude` still results in vendor being treated as `claude` for (a) PID resolution gating and (b) parent-exit session-end allowlist.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

PR Summary by Qodo

Prevent hosted sessions stuck "active" (watcher PID resolution + EndAgentSession retry)
🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Walkthroughs

Description

• Fix watcher parent-PID watchdog for Claude by resolving the durable agent PID via ancestry.
• Retry daemon EndAgentSession across transient SignalR reconnects without blocking stop
  termination.
• Add server-side reconciliation spec for sessions orphaned by hard daemon/host death.

Diagram

graph TD
  WM["WatcherManager"] --> PH["ProcessHelpers"] --> WC["WatchCommand"] --> S{{"kcap server"}}
  AO["AgentOrchestrator"] --> SC["ServerConnection"] --> CR["ConnectionRetry"] --> S

  subgraph Legend
    direction LR
    _mod["Module"] ~~~ _svc["Service"] ~~~ _ext{{"External"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Pass durable agent PID explicitly (env var/arg)

• ➕ Avoids OS-specific process-table introspection and edge cases
• ➕ Makes watchdog behavior deterministic and easier to reason about
• ➖ Requires changes across launch surfaces (agent wrappers/host integrations)
• ➖ Harder to retrofit for already-shipped hooks/sidecars

2. Shell out to ps/pgrep for (ppid, comm) lookup

• ➕ Simpler implementation than proc_pidinfo(/proc) parsing
• ➕ Potentially fewer unsafe/native calls
• ➖ Spawns subprocesses in hot paths (latency, failure modes, sandboxing)
• ➖ More platform variation in ps output; parsing becomes brittle

3. Fire-and-forget EndAgentSession on stop (don’t await)

• ➕ Avoids stop being blocked by reconnect entirely
• ➕ Minimal code changes
• ➖ Loses structured result handling (e.g., GenerateWhatsDone) unless separately re-plumbed
• ➖ Harder to test/observe failure; can hide persistent end-session failures

Recommendation: The PR’s approach is the best tradeoff: resolving the agent PID by walking PPID ancestry is robust to vendor-specific process-group topology (the root cause) while remaining unit-testable via an injected lookup. Reusing the existing ConnectionRetry helper for EndAgentSession keeps SignalR semantics consistent with RequestPermission and, by moving session-end to the post-exit finalize path, preserves fast user-initiated stop behavior without sacrificing reliability. The alternatives either require broader integration changes (explicit PID) or add new operational brittleness (ps parsing / fire-and-forget).

File Changes

Bug fix (5)

AgentOrchestrator.cs Stop agents without blocking on EndAgentSession retries +10/-16

Stop agents without blocking on EndAgentSession retries

• Removes the synchronous EndAgentSession call from HandleStopAgent to prevent stop from being delayed by reconnect/retry behavior. Relies on the read-loop finally path (FinalizeAgentRunAsync) as the reliable post-exit backstop for ending sessions and spawning what's-done generation.

src/Capacitor.Cli.Daemon/Services/AgentOrchestrator.cs

---

ServerConnection.cs Retry EndAgentSession across transient SignalR drops +11/-1

Retry EndAgentSession across transient SignalR drops

• Routes EndAgentSessionAsync through ConnectionRetry with a dedicated polling interval and retry logging. Ensures transient transport drops don’t permanently prevent session end for hosted agents.

src/Capacitor.Cli.Daemon/Services/ServerConnection.cs

---

WatchCommand.cs Make parent-exit watchdog decision explicit and always logged +63/-18

Make parent-exit watchdog decision explicit and always logged

• Introduces a unit-testable three-way decision (monitor / no parent / parent already dead) for the watcher’s parent-exit watchdog. Updates the watchdog startup path to loudly log skip cases, preventing silent failure modes that previously left watchers orphaned and sessions stuck active.

src/Capacitor.Cli/Commands/WatchCommand.cs

---

ProcessHelpers.cs Resolve durable coding-agent PID via PPID ancestry + per-OS process info +165/-1

Resolve durable coding-agent PID via PPID ancestry + per-OS process info

• Adds vendor-aware GetCodingAgentPid(vendor) that walks parent ancestry and matches the agent executable name (claude/codex), avoiding transient process-group/launcher PIDs. Implements Unix process inspection via macOS proc_pidinfo and Linux /proc/<pid>/stat parsing, plus strict name matching to avoid false positives.

src/Capacitor.Cli/ProcessHelpers.cs

---

WatcherManager.cs Use vendor-aware agent PID when spawning watchers +9/-6

Use vendor-aware agent PID when spawning watchers

• Switches watcher spawning to pass ProcessHelpers.GetCodingAgentPid(vendor) and expands comments explaining Claude vs Codex topology differences. This ensures the watcher monitors the correct long-lived agent process and can self-terminate + session-end on abrupt agent death.

src/Capacitor.Cli/WatcherManager.cs

---

Tests (4)

AgentOrchestratorVendorTests.cs Test stop termination is prompt even if EndAgentSession blocks +94/-2

Test stop termination is prompt even if EndAgentSession blocks

• Adds a regression test using a blocking EndAgentSessionAsync and a terminate-signaling PTY to prove HandleStopAgent still terminates promptly. Extends the CaptureServerConnection test double to record reasons and simulate reconnect-blocked session-end calls.

test/Capacitor.Cli.Tests.Unit/AgentOrchestratorVendorTests.cs

---

CodingAgentPidResolverTests.cs Add unit tests for ancestry-based agent PID resolver +131/-0

Add unit tests for ancestry-based agent PID resolver

• Introduces synthetic process-table tests covering Claude/Codex resolution, skipping transient launchers, nearest-ancestor selection, case-insensitive basename/path matching, non-matching substring safety, and max-hop/cycle defense.

test/Capacitor.Cli.Tests.Unit/CodingAgentPidResolverTests.cs

---

ParentWatchdogDecisionTests.cs Add unit tests for watcher watchdog decision outcomes +35/-0

Add unit tests for watcher watchdog decision outcomes

• Pins the three explicit watchdog outcomes to prevent silent skip regressions. Ensures null parent PID and dead-at-startup PID are surfaced distinctly from the monitor case.

test/Capacitor.Cli.Tests.Unit/ParentWatchdogDecisionTests.cs

---

ProcessHelpersTests.cs Add coverage for GetProcessInfo + vendor-aware PID fallback +43/-0

Add coverage for GetProcessInfo + vendor-aware PID fallback

• Adds Unix-only tests validating GetProcessInfo returns a non-empty comm and correct PPID for the current process, and that the vendor-aware GetCodingAgentPid overload still yields a live non-self PID when no matching agent exists in the ancestry.

test/Capacitor.Cli.Tests.Unit/ProcessHelpersTests.cs

---

Documentation (1)

2026-06-13-hosted-session-server-reconcile-spec.md Add spec for server-side reconciliation of orphaned active sessions +99/-0

Add spec for server-side reconciliation of orphaned active sessions

• Documents why client-side cleanup can’t handle hard daemon/host death and proposes two grace-windowed server triggers (daemon disconnect, watcher disconnect). Defines idempotency expectations and acceptance scenarios to ensure stuck sessions converge to a terminal state.

docs/superpowers/specs/2026-06-13-hosted-session-server-reconcile-spec.md

---