Skip to content

Terminal-session leak: builder respawns accumulate untracked shellper sessions that cleanup and the Builders view never see #1296

Description

@waleedkadous

Problem

Repeated respawns/resumes of a single builder each create a new shellper-backed terminal session without terminating the previous one, and only the newest session is linked to the builder record. Observed in a production workspace: 12 live sessions for one builder role (each a distinct shellper PID with a live .builder-start.sh child, aged ~3 days), plus a second builder at ×2.

Consequences (all observed)

  1. afx cleanup can't fully clean: it deregisters the builder, removes the worktree, and kills the one tracked session — the untracked ones survive as registered zombie PTYs whose bash loops point at a deleted worktree.
  2. The husk sweep never reaps them: they are registered (rows in terminal_sessions) with live children, so the unregistered+childless+aged predicate correctly skips them. Registered zombies are immortal.
  3. Dashboard confusion: the terminal list (one row per session) shows N duplicate rows; the Work/Builders view (keyed on the builders registry) shows nothing at all for a cleaned-up builder — the two views are both faithful to their sources and visibly disagree.
  4. Resource leak: N-1 idle bash loops + shellpers per builder, holding PTYs and memory. Tower restart recovery faithfully reconnects and re-persists all of them.

Asks

  1. Respawn should reuse-or-reap: when spawning/resuming a builder that already has a live session for the same role in the same workspace, either reattach to it or terminate it before creating the new one — never silently accumulate.
  2. Cleanup should sweep by role: afx cleanup should find and kill ALL sessions for the builder's role in that workspace, not just the linked one.
  3. A safe reap path for existing zombies: sessions whose builder record is gone AND whose cwd/worktree no longer exists are provably not live work — afx tower sweep-husks (or a sibling command) should be allowed to offer them for reaping, gated behind an explicit confirmation since the husk predicate deliberately cannot see them today.

Repro sketch

Spawn a builder; force several harness restarts/resumes over time; sqlite3 ~/.agent-farm/global.db "SELECT role_id, COUNT(*) FROM terminal_sessions GROUP BY role_id HAVING COUNT(*)>1" — duplicates accumulate; run afx cleanup for it — worktree and registry entry go, duplicate sessions remain.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/towerArea: Tower server / agent farm CLI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions