Tower/shellper: architect session-ID collisions + non-recovering crash loops after workspace restart

[waleedkadous]

Reported by an external adopter workspace (4 incidents in one session, 2026-07-22), after a full workspace restart: Tower restarted, 8 architects re-added via afx workspace add-architect. Details scrubbed; adopter can supply shellper PIDs/timestamps on request.

Symptom A — session-ID collision crash loop

An architect terminal crash-loops with:

Error: Session ID <uuid> is already in use

The shellper restarts its claude child forever; every child dies instantly. Two distinct holder cases observed:

1. Stale pre-restart shellper for the same architect name still held the session ID (survivor from before the Tower restart).
2. Unrelated foreground claude process on the user's own tty (started with wrong auth) had claimed the session ID.

Symptom B — wedged shellper / registry divergence

• After the holder is killed, the shellper does not recover — children keep dying within seconds even with the session free.
• The architect silently deregisters from Tower while its shellper process stays alive crash-looping. afx workspace remove-architect then fails with not found while the zombie process persists.
• A third architect silently vanished from the registry with no crash loop observed at all.

Recovery that worked (both collision cases)

1. Kill the wedged/stale shellper PID manually.
2. afx workspace add-architect --name <name> fresh.
3. Re-run /arch-init.

No state lost (state files + builders unaffected) — but the procedure is entirely manual and non-obvious.

Suggested fixes (from the reporter)

• (a) Mint-or-reclaim session IDs atomically on add-architect — never reuse a persisted session ID without verifying no live holder; or always mint fresh.
• (b) Crash-loop breaker in shellper — detect repeated instant-exit of the child (N failures in M seconds), give up loudly, and deregister cleanly instead of looping forever.
• (c) Make deregistration and process death atomic — today both divergence directions occur: a registry row without a live process, and a live process without a registry row.

Notes

• Related prior art: the verified-orphan shellper cleanup procedure (Orphaned shellper processes accumulate again (regression of #389): af cleanup / Tower restart leave untracked shellpers running #1007) exists precisely because shellper processes can outlive their registry rows; this report shows the inverse direction too, plus the session-ID reuse hazard that turns a stale survivor into an active denial of the architect slot.

[waleedkadous]

Forensic timeline from the reporting workspace (all times PDT, 2026-07-22; ~8 architects + ~15 builders; architect names anonymized as A/B/C).

Setup

• ~04:22 — full Tower restart. Main architect shellper up 04:22:41 (pid 37635).
• 04:24:07–08 — restore wave via afx workspace add-architect: six sibling shellpers (pids 39739, 39773, 39812, 39831, 39849, 39982 — including the three that later failed).
• 04:51:04 — one more architect added (pid 61274, session 89f3b4c0-a6d7-47cb-b71a-7cc38c0627a3).

Incident 1 — architect A (restored, then re-added): session-ID reuse collision with a dead sibling's remnant

• 05:21:41 — a sibling's afx send to A returned NOT_FOUND: first silent death — first-wave shellper 39982 already gone from the registry.
• ~05:21:59 — re-add of A spawned shellper pid 18907, whose args carried --session-id 166e1c25-fd66-46ed-a81c-5dfcf2295efd (persisted ID re-fed on respawn).
• ~06:10 — that terminal crash-looping on Session ID 166e1c25… already in use. At investigation time, 18907 was the only process referencing that session; A absent from the registry; A's builder alive.
• Inference: the re-add reused the persisted session ID while some remnant of the dead first-wave instance still held it (holder no longer identifiable by then). Plain kill of 18907 + fresh add-architect at 06:36 (pid 61353) → stable since.

Incident 2 — architect B: pure silent vanish

• First-wave shellper 39849 (started 04:24:08) simply gone by ~06:30 — no crash loop, no registry entry, no zombie. Re-add (pid 83869) → stable.

Incident 3 — architect C (richest data): foreign holder, then wedge-after-free

• ~06:56 — user accidentally started a foreign interactive claude on their own tty s119 (pid 97841, wrong auth), which claimed C's session 89f3b4c0…. C's registered shellper 61274 crash-looped with already in use.
• ~07:0x — user quit the foreign process. Key datapoint: with the session demonstrably free (zero processes referencing it), 61274 still did not recover — observed child pids 98525 then 166, each dying <8s after spawn, across 60+ seconds of watching.
• Meanwhile C had silently deregistered: afx workspace remove-architect C → not found while pid 61274 was alive and looping.
• ~07:12 — plain kill 61274 + add-architect → shellper 4599, child 4600 immediately stable (24s+ uptime at first check).

Pattern summary

• 2× session-ID reuse collision (one self-inflicted stale remnant, one foreign process) — consistent with session IDs being persisted per architect-name and re-fed to claude on respawn → fix (a).
• 1× wedge-after-free — crash loop persists with the session demonstrably free → fix (b) is the cure for the loop itself, and the wedge suggests the failure isn't (only) the session collision.
• 3× silent deregistration with process state diverged from the registry (both directions) → fix (c).

Reporter is willing to run live diagnostics on the next occurrence.

[waleedkadous]

On it! Working on a fix now.

[waleedkadous]

Live recurrence + recovery datapoint (maintainer's machine, 2026-07-22 ~14:00Z): the reporting workspace's default main architect hit the same pattern — deregistered from Tower while its original shellper (the pid from the timeline's 04:22 restart) was still alive and holding the session, ~9.5h after the workspace restart. Recovery differed from the named-architect cases: afx workspace add-architect --name main is refused ('reserved for the workspace's default architect'), so the path is: verify the zombie via the untracked-PID diff (running shellpers − GET /api/terminals), plain kill it, then re-run afx workspace start from the workspace root — which recreates the default architect (new shellper registered, child stable immediately, no crash loop). Fix (c) should account for the default architect having a different re-registration path than added architects.

[amrmelsayed]

Investigation findings from main architect (2026-07-23) — three datapoints for the in-flight fix (PR #1225)

1. Empirical: the collision does not reproduce on current claude (2.1.216) in interactive mode. Test on this machine: instance A launched with --session-id <uuid>, conversation materialized (jsonl on disk, holder alive and responding); instance B launched with the same --session-id, same cwd — B started cleanly to the REPL and answered prompts, no 'Session ID already in use' at startup or at message time. So the instant-exit collision the adopter hit is claude-version- or mode-dependent, not a current universal behavior. Implication: PR #1225's ps-scan guard is the right shape — it doesn't depend on claude's collision behavior at all — but the fix should not assume the error string/exit behavior as a stable contract for tests or detection. (Method: tmux + remain-on-exit; happy to share exact steps.)

2. A deregistered-but-alive shellper is never reaped by ANY existing path — relevant to fix (c) scope. The Symptom-B zombie (live shellper, no registry row) survives even the next Tower restart: killOrphanedShellpers (session-manager.ts) probes each candidate's socket and skips kill when the socket is responsive ('reality trumps SQLite' — correct for the #1198 incident it was hardened against, but it means a healthy-socket zombie whose rows were deleted is permanently invisible to the sweeper). shellper-main.ts meanwhile deliberately stays alive after child exit waiting for a SPAWN frame that, post-deregistration, never comes. So both divergence directions currently have no automated reconciliation; fix (c) needs an explicit 'live shellper with no registry row → adopt or terminate' decision somewhere.

3. The crash-loop breaker (#1149) has two coverage gaps in this exact scenario. (i) crashLoopFallback is only wired when resumed && storedSessionId && fallback — a fresh-mint launch gets none (defensible today, but worth noting if (b) is revisited). (ii) The breaker counts only exit.code !== 0 exits — if any claude version/mode exits 0 on the collision (unverified; see datapoint 1's version-dependence), the breaker is blind to it while restartCount still burns toward maxRestarts, after which removeDeadSession + the ptySession exit handler produce exactly the Symptom-B divergence (row deleted, process alive). Same exit-code-semantics territory as #1214.

Also for the record: #1224 is orthogonal to the recently merged #1198/#1204 (reconnect-error swallowing) and #1215/#1218 (replay-wait stall) — launch-args layer vs connection/replay layers; neither caused nor mitigates this.