[AI-839] Fix daemon PID file identity check on Linux + detached stdio hang

[alexeyzimarev]

Problem (AI-839)

A live, server-registered daemon could not be managed from the CLI on Linux: status reported not running (stale PID file), doctor showed HELD … (no pid file), stop --yes was a no-op, and start -d refused with "already running under the name" — leaving no CLI path to restart it. kcap daemon start -d also hung instead of returning.

Root cause

The lifecycle decides "is the process at this PID still our daemon?" by comparing Process.StartTime for exact tick-equality. On Linux that value is recomputed per process from a boot-time estimate (wall clock − uptime), so the daemon and a later status/stop/doctor invocation disagree by ms-to-seconds (measured ~0.8 ms in a Linux container). Every live daemon was therefore misread as a stale PID file:

• status → "stale" and deletes the pid file
• doctor → then shows (no pid file) while the flock is genuinely HELD
• stop → no-op
• start -d → re-spawns, new daemon collides on the flock → "already running under the name"

macOS uses an absolute sysctl start timestamp that is identical across processes, so it never reproduced.

Secondary: StartDetached redirected only stdin, so the detached daemon inherited the caller's stdout/stderr and held the pipe open for life → start -d hung for any output-capturing caller.

Fix

• ProcessStartToken (Core) — a cross-process-stable start token. Linux reads /proc/<pid>/stat field 22 (kernel boot-relative starttime, byte-identical for every reader, immune to clock drift); macOS/Windows keep Process.StartTime.Ticks. Used by DaemonLock.WritePidFile, the CLI WritePidFile/ReadPidFile/IsOurDaemon. A token mismatch stays conclusive (rejects recycled PIDs); the name fallback only applies when the token is unreadable.
• Detached stdio — redirect all three streams, PreventInheritedStdHandles() (Windows), close them parent-side (same AI-820 idiom as the watcher / what's-done spawns).

Verification (real Linux, .NET SDK container)

• start -d → status running → duplicate start -d correctly refused → stop kills it → status clean → doctor STALE/cleanable.
• kcap daemon start -d | cat returns in ~1 s (was a hang).
• New regression tests pass on macOS and Linux (the /proc assertion runs on Linux); full unit suite 1412/1412; both AOT publishes IL-warning clean.

🤖 Generated with Claude Code

[linear-code]

AI-839

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0)

1. Legacy token blocks stop ✓ Resolved 🐞 Bug ≡ Correctness

Description

DaemonCommands.IsOurDaemon treats any readable start-token mismatch as authoritative and does not
fall back to the process-name check, so any existing PID file with a different second-line format
will be treated as stale even if the daemon is still alive. On Linux upgrades, this can strand an
already-running (pre-AI-839) daemon because status/stop will classify it as stale and delete its PID
file.

Code

src/Capacitor.Cli/Commands/DaemonCommands.cs[R556-561]

+            if (expectedStartToken is not null) {
+                var actual = ProcessStartToken.ForPid(pid);
+
+                // Token readable → authoritative. Unreadable → name fallback.
+                if (actual is not null) return actual == expectedStartToken;
            }

Evidence

The PID file second line is read verbatim and passed into an identity check that treats any readable
token mismatch as definitive; stale classification triggers PID file deletion in status/stop.
Meanwhile Linux tokens now come from /proc/<pid>/stat starttime, so older PID-file second-line
values from a different scheme will never match and will be treated as stale.

src/Capacitor.Cli/Commands/DaemonCommands.cs[523-561]
src/Capacitor.Cli/Commands/DaemonCommands.cs[310-369]
src/Capacitor.Cli.Core/ProcessStartToken.cs[39-68]
src/Capacitor.Cli.Core/DaemonLockPaths.cs[61-67]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`IsOurDaemon` returns immediately on token equality when both the PID file token and the runtime token are readable. This makes token mismatches conclusive, which breaks upgrades when a running daemon was started by an older version that wrote a different token scheme on the PID file’s second line.

### Issue Context
- `ReadPidFile` treats the second line as an opaque string token.
- `IsOurDaemon` treats readable token comparison as authoritative and does not fall back on mismatch.
- `status` / `stop` delete the PID file when `IsOurDaemon` returns false.

### Fix Focus Areas
- Add explicit token scheme/version so mismatches are only conclusive within the same scheme (e.g. `linux:<bootId>:<starttime>` vs `ticks:<ticks>`).
- When the PID file’s second line is unversioned/legacy (no prefix), treat it as “no token” on Linux so `IsOurDaemon` uses the name fallback, preserving manageability across upgrades.
- Optionally: update stale comment about PID file containing StartTicks.

- src/Capacitor.Cli/Commands/DaemonCommands.cs[523-575]
- src/Capacitor.Cli.Core/ProcessStartToken.cs[39-68]
- src/Capacitor.Cli.Core/DaemonLockPaths.cs[61-67]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Detached stderr/out may fail 🐞 Bug ☼ Reliability

Description

StartDetached redirects stdout/stderr and closes the parent ends immediately; if the daemon writes
to Console.Out/Console.Error (e.g. lock/name-in-use errors), those writes can fail (broken
pipe/IOExceptions) and potentially alter failure behavior/exit code. This is most likely to surface
on detached-start failure paths where the daemon currently emits user-facing Console.Error messages.

Code

src/Capacitor.Cli/Commands/DaemonCommands.cs[R171-187]

+        // Redirect ALL three standard streams so the detached daemon does not
+        // inherit our stdout/stderr (AI-839). Left un-redirected, the daemon
+        // keeps the terminal — or a capturing parent's pipe — open for its
+        // whole lifetime, so `kcap daemon start -d` appears to hang: it returns
+        // and the daemon reparents to init, but anything reading our piped
+        // output blocks on EOF that never comes. The daemon logs to --log-file,
+        // so it has no use for these streams. (Same pipe-leak hazard and fix as
+        // the AI-820 watcher / what's-done spawns.)
        var psi = new ProcessStartInfo {
            FileName               = daemonPath,
            UseShellExecute        = false,
            RedirectStandardInput  = true,
-            RedirectStandardOutput = false,
-            RedirectStandardError  = false,
+            RedirectStandardOutput = true,
+            RedirectStandardError  = true,
            CreateNoWindow         = true
        };
        psi.ArgumentList.Add("--log-file");

Evidence

The detached start path explicitly redirects stdout/stderr to pipes and closes them immediately,
while the daemon still emits Console.Error messages on common early-exit paths (like lock
contention). With no reader on the pipes, these writes can fail and affect behavior.

src/Capacitor.Cli/Commands/DaemonCommands.cs[171-206]
src/Capacitor.Cli.Daemon/DaemonRunner.cs[116-137]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
In detached mode, the daemon’s stdout/stderr are redirected to pipes and the parent closes its read ends right away. Any daemon-side `Console.*` writes after detachment can fail and potentially change failure-path behavior.

### Issue Context
The daemon still uses `Console.Error.WriteLineAsync(...)` for certain startup failures (e.g. name-in-use), which is precisely when detached start is most likely to exit early.

### Fix Focus Areas
- Prefer not writing to Console at all when `--log-file` is set (detached mode). Early in daemon startup (immediately after parsing `--log-file`), set `Console.SetOut(TextWriter.Null)` / `Console.SetError(TextWriter.Null)` or wrap console writes in best-effort try/catch for IO failures.
- Alternatively, implement true null-device redirection for detached processes (platform-specific) rather than redirect-to-pipe-then-close.

- src/Capacitor.Cli/Commands/DaemonCommands.cs[171-206]
- src/Capacitor.Cli.Daemon/DaemonRunner.cs[116-137]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Linux token lacks boot ID ✓ Resolved 🐞 Bug ☼ Reliability

Description

On Linux the start token is only /proc/<pid>/stat starttime (ticks since boot), so it is not
globally unique across reboots. If a stale PID file survives a reboot, a very rare PID+starttime
coincidence could cause IsOurDaemon to accept a different process as the daemon.

Code

src/Capacitor.Cli.Core/ProcessStartToken.cs[R58-67]

+    static string? ReadLinuxStartTicks(int pid) {
+        try {
+            var stat       = File.ReadAllText($"/proc/{pid}/stat");
+            var afterComm  = stat.LastIndexOf(')');
+
+            if (afterComm < 0 || afterComm + 2 >= stat.Length) return null;
+
+            var fields = stat[(afterComm + 2)..].Split(' ', StringSplitOptions.RemoveEmptyEntries);
+
+            return fields.Length > 19 ? fields[19] : null;

Evidence

The implementation and its own documentation confirm the Linux token is clock ticks since boot, and
the daemon identity check treats token equality as the strongest/authoritative match when readable,
so adding a boot identifier would eliminate the (low-likelihood) cross-boot ambiguity.

src/Capacitor.Cli.Core/ProcessStartToken.cs[22-27]
src/Capacitor.Cli.Core/ProcessStartToken.cs[58-67]
src/Capacitor.Cli/Commands/DaemonCommands.cs[552-561]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Linux tokens use only boot-relative `starttime` (jiffies since boot). Adding a boot identifier would make the token unambiguous across reboots.

### Issue Context
`IsOurDaemon` treats a readable token match as authoritative.

### Fix Focus Areas
- Extend the Linux token format to include a boot identifier (e.g. `/proc/sys/kernel/random/boot_id` or `btime`) alongside `starttime`.
- Consider introducing explicit scheme prefixes at the same time (also helps backward compatibility).

- src/Capacitor.Cli.Core/ProcessStartToken.cs[22-27]
- src/Capacitor.Cli.Core/ProcessStartToken.cs[58-67]
- src/Capacitor.Cli/Commands/DaemonCommands.cs[552-561]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

PR Summary by Qodo

Fix Linux daemon PID identity checks and prevent detached stdio hangs
🐞 Bug fix 🧪 Tests 🕐 20-40 Minutes

Walkthroughs

Description

• Introduce a cross-process stable start token to validate daemon PID files on Linux.
• Update daemon CLI lifecycle checks (start/status/stop/doctor) to use the new token.
• Redirect and close all stdio streams for detached daemon starts to avoid pipe hangs.

Diagram

graph TD
  cli["kcap CLI"] --> pid[("PID file")] --> check["IsOurDaemon()"] --> token["ProcessStartToken"]
  token --> proc{{"/proc/<pid>/stat"}}
  token --> st{{"Process.StartTime"}}
  cli --> detach["StartDetached()"] --> stdio["Redirect+close stdio"] --> daemon["kcap-daemon"]

  subgraph Legend
    direction LR
    _p["Process/component"] ~~~ _f[("File")] ~~~ _e{{"OS API"}}
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Compare Process.StartTime with tolerance on Linux

• ➕ Minimal code churn; keep existing PID file format
• ➖ Still vulnerable to clock drift/NTP adjustments and per-call boot-time estimation error
• ➖ Tolerance-based checks increase PID-reuse false positives/negatives and are hard to tune

2. Rely solely on the per-name flock (no PID identity check)

• ➕ Avoids PID/token complexity entirely
• ➖ CLI would need a different mechanism to map a lock holder back to a PID for stop/status
• ➖ Harder to implement robust stop semantics without OS-specific handle introspection

Recommendation: The PR’s approach (persist a stable, kernel-derived start token on Linux and compare it authoritatively when readable) is the most reliable way to distinguish a live daemon from PID reuse across separate CLI invocations. The detached stdio fix is also the correct strategy for avoiding pipe hangs when the caller captures output.

File Changes

Bug fix (3)

ProcessStartToken.cs Add cross-process stable process start token utility +72/-0

Add cross-process stable process start token utility

• Introduces ProcessStartToken for identifying a specific process incarnation across reader processes. Uses /proc/<pid>/stat starttime on Linux and Process.StartTime ticks on macOS/Windows, returning null when unreadable to enable safe fallbacks.

src/Capacitor.Cli.Core/ProcessStartToken.cs

---

DaemonLock.cs Write daemon PID files with stable start token +7/-12

Write daemon PID files with stable start token

• Replaces StartTime tick persistence with ProcessStartToken in the daemon-side PID file writer. Ensures PID files remain verifiable by separate CLI processes on Linux, while preserving PID-only fallback behavior when the token is unavailable.

src/Capacitor.Cli.Daemon/DaemonLock.cs

---

DaemonCommands.cs Use start token for daemon identity; fix detached stdio inheritance +48/-36

Use start token for daemon identity; fix detached stdio inheritance

• Updates PID file parsing and daemon identity verification to compare stable start tokens (and only fall back to process name when the token is missing/unreadable). Also redirects stdout/stderr for detached starts and closes all three redirected streams to prevent output-capturing callers from hanging.

src/Capacitor.Cli/Commands/DaemonCommands.cs

---

Tests (2)

DaemonLockTests.cs Add regression test for PID file token contents +29/-0

Add regression test for PID file token contents

• Adds a unit test asserting the daemon lock writes a PID file containing exactly two lines: current PID and the computed stable start token. This guards the Linux stale-PID misclassification regression.

test/Capacitor.Cli.Tests.Unit/Daemon/DaemonLockTests.cs

---

ProcessStartTokenTests.cs Add unit tests for ProcessStartToken behavior (including Linux /proc) +64/-0

Add unit tests for ProcessStartToken behavior (including Linux /proc)

• Adds coverage for token non-emptiness, self-consistency (ForCurrent vs ForPid), nonexistent PID behavior, and a Linux-specific assertion that the token matches /proc/<pid>/stat field 22. Protects against regressions back to Process.StartTime-based tokens on Linux.

test/Capacitor.Cli.Tests.Unit/Daemon/ProcessStartTokenTests.cs

---

[alexeyzimarev]

Addressed the qodo review in 00c5212:

1. Legacy token blocks stop (Action required) — Fixed. Tokens are now scheme-prefixed and ProcessStartToken.Matches is tri-state: same-scheme mismatch stays conclusive (recycled-PID rejection), while a legacy/foreign or unreadable token returns null so IsOurDaemon falls back to the image-name check. Verified on Linux: a live daemon with a pre-AI-839 bare-ticks PID file now reports running and stop kills it instead of stranding it + deleting the PID file.

3. Linux token lacks boot ID (Optional) — Fixed in the same change. Linux token is now lx:<boot_id>:<starttime> (/proc/sys/kernel/random/boot_id); macOS/Windows use tk:<absolute-ticks>, which is already unambiguous across reboots.

2. Detached stderr/out may fail (Recommended) — Not reproducible, leaving as-is. I exercised a detached early-exit path (start -d --max-agents notanumber, which writes to Console.Error after the parent closed the pipe read-ends) on Linux: the daemon still exits with its intended code 1 and the CLI reports it correctly. .NET tolerates the broken-pipe console write, so failure behavior/exit code is unchanged. The late death-rattle writes are already wrapped in try/catch. Happy to add Console.SetError(TextWriter.Null) under --log-file if you'd still prefer the defensive belt-and-braces, but it isn't needed for correctness.