Shellper process leak: orphaned processes accumulate across sessions

[waleedkadous]

Bug Description

Shellper processes (shellper-main.js) are not being killed when their parent terminal sessions end. This causes orphaned processes to accumulate, eventually exhausting system resources and causing posix_spawnp failed errors that block new terminal creation.

Found ~193 orphaned shellper processes on a development machine after a day of builder activity.

Root Cause Analysis

Three gaps in the shellper cleanup lifecycle:

Gap 1: killTerminalWithShellper() exists but isn't wired into cleanup

tower-instances.ts has killTerminalWithShellper() that properly kills both PtySession AND the underlying shellper process. But af cleanup (cleanup.ts:162-171) calls TowerClient.killTerminal() which goes through a path that only kills the PtySession wrapper — the shellper process survives.

Gap 2: PtySession exit handler doesn't kill shellper

When a shellper-backed session exits (e.g., builder context exhaustion), tower-instances.ts:404-414 removes from cache and SQLite but never calls shellperManager.killSession(). The shellper process stays alive indefinitely.

Gap 3: Tower shutdown preserves ALL shellper processes indiscriminately

tower-server.ts:112-149 deliberately skips killing shellpers so architect sessions survive restarts. But builder shellpers also survive — and accumulate across restarts.

Root Design Issue

Shellper processes are spawned with detached: true and child.unref() (session-manager.ts:160-184). This is correct for architects (survive Tower restarts) but wrong for builders (should die with their session). There's no distinction between architect and builder shellper sessions in the cleanup logic.

Symptoms

• posix_spawnp failed errors when spawning new builders
• System accumulates hundreds of orphaned Node.js processes
• Only fixable by manually pkill -f shellper-main (excluding active PIDs)

Expected Fix

1. Wire killTerminalWithShellper() into the DELETE endpoint and af cleanup flow
2. Kill shellper on PtySession exit for non-architect sessions
3. On Tower shutdown, kill builder shellpers but preserve architect ones
4. Ensure test teardown in session-manager tests kills all spawned shellpers

Key Files

• packages/codev/src/terminal/session-manager.ts:160-184 (spawn)
• packages/codev/src/agent-farm/servers/tower-instances.ts:404-414 (exit handler)
• packages/codev/src/agent-farm/servers/tower-instances.ts:471-482 (killTerminalWithShellper)
• packages/codev/src/agent-farm/servers/tower-server.ts:112-149 (shutdown)
• packages/codev/src/agent-farm/commands/cleanup.ts:162-171 (af cleanup)
• packages/codev/src/terminal/__tests__/session-manager.test.ts:76-83 (test teardown)

[waleedkadous]

Fixed by PR #300 (Spec 0116 — Shellper resource leakage prevention).