Skip to content

browse: CLI kills a live-but-busy daemon during heavy page loads — ~7s health-probe window too short (follow-up to #1781); all session state lost #2219

Closed

Description

@DavidCnoops

Environment

  • gstack: 1.58.4.0 (9fd03fa), CLI built from same tree
  • OS: macOS 26.3.1 (arm64), Bun 1.3.9
  • Playwright: chromium_headless_shell-1208
  • Target app: local Next.js 15 dev server (heavy routes, cold compiles routinely 15–60s)

Symptom

During a /browse-driven QA session the headless daemon "randomly" dies and gets replaced 4 times in ~15 minutes. Every replacement loses all session state (login cookies, viewport, tabs), forcing a re-login each time. Nothing in the visible CLI output explains it — the daemon is spawned with stdio: 'ignore', and the [browse] Server connection lost. Restarting... / [browse] Starting server... lines are easy to lose when output is filtered (e.g. | tail -1).

Root cause (evidenced, reproduced 3× with daemon stderr captured)

The daemon never crashes on its own. The CLI kills it.

Chain, with refs at v1.58.4.0:

  1. A goto to a heavy dev route exceeds the 15s navigation timeout. Playwright throws, the daemon responds normally (audit log: goto … status=error durationMs=15010) — but the navigation keeps loading in Chromium.
  2. The next CLI command arrives while the daemon/Chromium is still churning (CDP console/network event flood from the dev app). The fetch fails at connection level, and probeHealthWithBackoff (browse/src/cli.ts:251 — 3 attempts × 250ms backoff, 2s timeout each, so roughly a 7s grace window) also fails.
  3. sendCommand's catch (browse/src/cli.ts:586-605) declares the daemon dead: killServer(oldState.pid) → SIGTERM → daemon replaced → all cookies/login lost (ephemeral profile).
  4. The daemon's death is then silent/garbled for two separate reasons filed separately: SIGTERM "ignore" being cosmetic (see companion issue) and the browser?.process() TypeError in resolveDisconnectCause ([Windows] browse fails to start: Chromium disconnects on launch, then resolveDisconnectCause crashes with browser?.process is not a function #2085, secondary problem — confirmed on macOS, comment there).

Captured daemon stderr at the moment of death (identical in all 3 reproductions):

[browse] Received SIGTERM (ignoring — use /stop or Ctrl+C for intentional shutdown)
[browse] Tab closed (id=1, remaining=0)
[browse] FATAL unhandled rejection: browser?.process is not a function. (In 'browser?.process()', 'browser?.process' is undefined)

Server-side audit log around one of the original deaths — note the daemon fully served the command, then was killed seconds later (next entry is the freshly spawned replacement):

15:11:11.135Z | viewport 1440x900 | ok  | dur: 27
15:11:26.233Z | goto /en/training | error | dur: 15010   <- daemon responded fine
15:11:37.532Z | screenshot        | ok  | dur: 34        <- answered by the REPLACEMENT daemon

Why #1781's protections don't cover this

The #1781 fixes are present and working: the 30s AbortError path refuses to kill ("busy, not restarting"), and the connection-error path probes health before killing. But the probe window (~7s max) is shorter than how long a daemon stays unresponsive while Chromium loads a heavy dev-mode page with a timed-out navigation still in flight. So the "busy, not dead" case #1781 was designed to protect still ends in kill+restart — just via the connection-error branch instead of the AbortError branch.

Ruled out with evidence: kernel/jetsam kills (unified log clean), macOS crash reports (none), harness/session teardown (daemon is correctly detached: PPID 1, own pgid — and a manually session-isolated daemon died identically), killOrphanChromium (headed-profile lock only), terminal-agent watchdog (kills agents by identity record only), idle timeout.

Repro (deterministic-ish, no special app needed)

  1. Serve any route that takes >15s to finish loading (a cold-compiling Next.js dev route, or an endpoint that accepts and stalls).
  2. Spawn the daemon with stderr captured: BROWSE_PARENT_PID=0 bun run browse/src/server.ts 2>&1 | tee /tmp/browse.log
  3. browse goto <slow-route> → "Timeout 15000ms exceeded" (daemon healthy, nav still in flight).
  4. Within ~5s, run any other browse command (browse url).
  5. Observe: SIGTERM → tab closed → FATAL in the log; a fresh daemon answers the command; all session state gone.

Timing-dependent (needs the daemon to actually blow the ~7s probe window), so it fires reliably on heavy real pages and intermittently on synthetic ones. In my sessions it fired 4/4 times on cold dev routes.

Suggested direction

  • Lengthen/adapt the probe window when the last command was a navigation that timed out (the highest-probability "busy" state), or track "nav in flight" server-side and have /health report busy-but-alive so the CLI can distinguish.
  • Consider reconnect-don't-respawn: if the PID is alive, keep waiting/retrying rather than killing — a lost session is strictly worse than a slow command for QA flows.
  • On unavoidable restart, restore state automatically (the state save/load machinery already exists) so a kill doesn't cost the login.

Activity

  1. xg-gh-25 commented on Jul 10, 2026

    @xg-gh-25

    This is a health-probe timeout inversion — the daemon is alive but legitimately busy, yet gets killed because the probe window is shorter than the busy period.

    Core issue:

    The 7s probe window (3 attempts × 250ms backoff, 2s timeout each) was sized for "daemon crashed" detection, not "daemon is doing real work" scenarios. A Chromium navigation that times out at 15s leaves the browser process churning (CDP event flood, compilation, network) for significantly longer than 7s. From the CLI's perspective this is indistinguishable from a dead daemon, so it kills and respawns — losing all session state (cookies, login, tabs).

    Why #1781's protections don't cover this:

    The 30s AbortError path correctly refuses to kill during timeout, but the connection-error path (which fires when the next command arrives while the daemon is still busy) only gets the ~7s probe window before declaring death. So the "busy-but-alive" case #1781 was designed to protect still ends in kill+restart, just via a different branch.

    Production pattern we use:

    In SwarmAI's long-running daemon layer (similar problem: background ingestion/indexing that blocks health probes), we solved this with stateful health checks:

    1. The daemon tracks its own "busy" state (e.g., {"status": "busy", "task": "navigation", "started_at": <timestamp>}) and returns that in /health responses instead of a simple 200/timeout.

    2. The CLI interprets "busy" as "alive but don't send new commands" — it retries the health check with a longer backoff (30s+) instead of killing, and surfaces the busy state to the user ("Waiting for navigation to complete...").

    3. If the daemon is actually dead, the connection fails entirely (no response, not even "busy"), so the CLI can distinguish true death from legitimate busy-ness.

    Alternate approach (simpler, no protocol change):

    Lengthen the probe window dynamically when the last command was a navigation timeout:

    const lastCmdWasNavTimeout = oldState.lastError?.includes("Timeout 15000ms exceeded");
    const probeAttempts = lastCmdWasNavTimeout ? 6 : 3; // 30s window for nav timeouts
    const probeBackoff = lastCmdWasNavTimeout ? 5000 : 250;

    This keeps the existing health endpoint simple (just responds or doesn't) while giving legitimately-busy daemons more time to recover.

    On the SIGTERM "ignore" cosmetic issue:

    The daemon logs "ignoring SIGTERM" but then exits anyway. This is a separate logging bug (the ignore handler doesn't actually prevent the default termination), but it's orthogonal to the root cause here. Even if SIGTERM handling were fixed, the core problem remains: the CLI shouldn't be sending SIGTERM when the daemon is busy-but-alive.

    Minimum fix:

    Your suggested direction of "recognize nav-in-flight and adapt probe window" is the right path. The resolveDisconnectCause TypeError (#2085) is a secondary issue — it obscures the real cause of death in logs, but doesn't cause the kill itself.

    For QA flows where losing a login costs 5+ minutes, the reconnect-don't-respawn approach (keep waiting if the PID is alive) is worth considering as a user-level flag: --preserve-session that never kills, only waits.


    This analysis draws from SwarmAI's daemon health-check patterns in production. SwarmAI. Discussion: T-MEM

  2. time-attack commented on Jul 15, 2026

    @time-attack
    Contributor

    @16francej — recommend closing this issue only after a corrected #2231 lands. It must preserve live-but-busy sessions during the heavy-navigation reproduction while safely recovering genuinely wedged daemons and passing expanded probe coverage. Preserve live-process confirmation, session-state checks, and a separate wedged-process control.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions