Skip to content

fix: reconcile stale background-task registry entries (phantom parked dot) - #435

Open
serxa wants to merge 2 commits into
mainfrom
serxa/phantom-parked-fix
Open

serxa wants to merge 2 commits into
mainfrom
serxa/phantom-parked-fix

Conversation

@serxa

@serxa serxa commented Sep 12, 2026 •

Copy link
Copy Markdown
Member

What this fixes

A session sometimes shows a permanent pulsing violet "parked" dot in the sidebar even though it is idle with nothing running. It never clears on its own — a page refresh doesn't help (the flag is server-side); only a daemon restart does.

Why it happens

The daemon tracks a session's background tasks (run_in_background Bash/Agent, Monitor, Workflow) in an in-memory registry built from the CLI's event stream: an entry goes running on task_started and only clears when a matching completion event arrives. has_background_tasks — which lights the dot — is "any entry still running".

If that completion event never arrives, the entry is stuck running forever. That happens when the CLI stops observing a task it started (e.g. a nohup … & child detached inside a run_in_background call, or a dropped event), or when the client is torn down before the task finishes.

It is self-perpetuating: the idle-client sweep deliberately skips sessions that have a live background task (so it doesn't kill the watcher that delivers a real task's completion turn). A stuck entry makes the sweep skip the session forever → its client is never torn down → nothing ever reconciles the entry. The registry is in-memory, so only a restart clears it.

Impact is cosmetic (the dot) plus a CLI subprocess that is never reaped — no effect on routing or replies.

The fix

  • Reconcile on teardown. When a session's client goes away, any entry still "running" is marked done and the UI is refreshed — its completion can never arrive once the client is gone. This runs on these client-removal paths: idle sweep, idle-stream watcher, one-shot runs, and the dead-client / model-switch / transport-retry / crash / cancel / error paths.
  • Time-box a silent entry. An entry that has produced no event for longer than sessions.bg_task_stale_minutes (default 24h; 0 disables) is treated as no-longer-live, so the sweep stops skipping the session and reconciles it. The window sits far above any real silent job — a from-scratch build or a long test/fuzzer run can emit nothing for hours — so a genuinely-running task is never reaped early (which would kill its watcher and drop the completion turn).
  • Never disconnect a busy client. The idle sweep re-checks at teardown and backs off if a new turn started on — or replaced — the client while it was working.

Orphaned Workflow-panel state settles the same way. No frontend change: the existing session_running broadcast already carries the flag that drives the dot.

Testing

tests/test_bg_task_registry.py (14 cases) covers teardown reconcile + broadcast, the staleness cutoff (stale / fresh / disabled), the sweep's skip-vs-reap behaviour, event-refresh liveness, the busy-client guard, Workflow settling, and unchanged normal completion. Full suite passes.

Not in scope (pre-existing)

Two lifecycle weaknesses are left as-is (can follow up separately): the idle sweep isn't under a global lifecycle lock (only the targeted busy-client guard added here), and on a normal completion the sidebar flag refreshes on the next turn/refresh rather than on the terminal event.

serxa and others added 2 commits September 12, 2026 15:37
A session could show a permanent pulsing "parked" dot in the sidebar even
when idle with no background work. The in-memory _bg_task_registry is
event-driven: an entry goes "running" on task_started and only settles on a
terminal task_updated/task_notification. If that terminal event never
arrives — a detached `&` child the CLI stops observing, a dropped event, or a
client torn down before completion — the entry stays "running" forever.
Nothing reconciles it: has_background_tasks stays true, and the idle-client
sweep skips the session (to preserve the watcher that delivers a real task's
completion), so the client is never discarded and no teardown runs. Within a
daemon lifetime the dot is permanent — only a restart rebuilds the registry.

Two defences:

1. Reconcile on teardown. _discard_client now terminalizes any still-running
   entries (the delivering client and its idle-stream watcher are gone, so a
   completion event can never arrive), prunes them, and broadcasts
   background_tasks_update + session_running so the sidebar dot clears live
   instead of at the next restart. This is the path the idle sweep, the
   idle-stream watcher, and one-shot runs all funnel through.

2. Staleness trap-breaker. _has_live_background_tasks treats a "running" entry
   with no event for longer than sessions.bg_task_stale_minutes (default 360)
   as not-live, so the idle sweep can proceed and reconcile it. Each entry now
   carries last_event_at, refreshed on every task_* event, so an actively
   progressing task never goes stale. The window defaults well above the
   longest legitimately-silent background task (a from-scratch build or a long
   test suite emits no events for an hour or more) so a genuinely-running
   run_in_background task is never reaped early; 0 disables the check.

Adds tests/test_bg_task_registry.py covering the teardown reconcile, the
staleness predicate, the idle-sweep skip/reap paths, event-refresh liveness,
and the unchanged normal-completion path.
Addresses the independent review of the phantom-parked-dot fix:

- Frontend contract: reconcile marks orphaned entries "done" (was "stopped",
  which is outside the background_tasks_update union running|done|failed|
  timeout and off the registry's terminal vocabulary — the CLI "stopped"
  outcome already maps to "done").
- Reconcile on every client removal, not just _discard_client: the
  dead-client / model-switch / transport-retry / crash-retry paths now go
  through a shared _teardown_live_client helper, and the cancel/error turn
  teardowns reconcile too — so an orphaned entry can't be left with no client
  for the idle sweep to reach.
- Idle-sweep teardown race: _discard_client(only_if_idle=True) aborts if a new
  turn reused or replaced the client during the reconcile/memorize awaits, so
  the sweep never disconnects a client an active turn is using.
- Settle orphaned Workflow snapshots too: the reconcile terminalizes,
  persists, broadcasts and prunes still-running _workflows snapshots so a
  workflow panel spinner clears alongside the sidebar dot.
- Raise the staleness default 6h -> 24h so a genuinely long, silent
  background build/test is never reaped early.

Tests: 4 new cases in tests/test_bg_task_registry.py (teardown-live-client
reconcile+detach, only_if_idle abort + proceed, workflow settle).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@serxa
serxa marked this pull request as ready for review September 17, 2026 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant