afx: reconcileTerminalSessions doesn't self-repair orphaned architect registry rows (inverse of #1150)

[amrmelsayed]

Problem

Inverse of #1150. When Tower boot reconciliation finds a live type='architect' terminal_sessions row whose matching architects registry row is missing, it reconnects the terminal (correctly — the process is alive) but does NOT re-create the registry row. The architect appears to work for the current Tower lifetime, then silently fails in two ways:

1. Doesn't survive afx workspace stop/start: launchInstance's reconciliation loop iterates getArchitects(workspacePath) to respawn siblings. No row → no respawn → architect disappears on next start with no warning.
2. Loses conversation on in-process crash: session-manager.ts's auto-restart bakes --resume <session_id> from the architect row. No row → no stored session id → fresh spawn without resume → new conversation, prior context lost.

Both failures are silent to the user — the architect worked, then didn't, with no diagnostic pointing at the missing registry row.

How the state gets orphaned:

• Partial removal (e.g., removeArchitect succeeds at killing the shellper but fails at deleting terminal_sessions — inverse of afx: removed sibling architect resurrects after Tower recovery — reconciliation trusts persisted row without liveness check #1150, which was DB delete succeeding but shellper survival)
• Exit-handler race (row deleted between reconcile pass reading it and terminal exit handler pruning it)
• WAL loss on OS crash (same class as afx: removed sibling architect resurrects after Tower recovery — reconciliation trusts persisted row without liveness check #1150's RC3.2 concern — recently-committed transactions can be lost with synchronous = NORMAL)

Origin

Follow-up filed by the pir-1150 builder during its dev-approval discussion. #1150 fixes the "row survives, terminal doesn't → zombie respawn from stale row" direction. This issue fixes the mirror direction: "terminal survives, row doesn't → silent loss on next lifecycle event."

Root cause hypothesis

reconcileTerminalSessions in packages/codev/src/agent-farm/servers/tower-terminals.ts reconnects each live type='architect' terminal but assumes the accompanying architect registry row exists. There is no cross-check that verifies the row is present or a self-repair path that re-creates it.

Fix sketch

In reconcileTerminalSessions Phase 1 (tower-terminals.ts), when reconnecting a type='architect' terminal_sessions row:

1. Check getArchitectByName(workspacePath, session.role_id) — if a row exists, no action needed
2. If missing, re-create it via setArchitectByName(workspacePath, session.role_id, null) (or the appropriate helper)
   • workspace_path: from the terminal_sessions row
   • name: from role_id (which is how architects register their name)
   • session_id: null (backfills at next restart-bake once the process crashes-then-recovers and a fresh session id is minted)
3. Log INFO: "self-repaired architect registry row for <workspace>/<name> (was orphaned from terminal_sessions)"

This is the inverse operation of #1150's fix: #1150 prunes registry rows without live terminals; this issue re-creates registry rows for terminals that survived.

Scope

• packages/codev/src/agent-farm/servers/tower-terminals.ts — reconcileTerminalSessions Phase 1 self-repair branch
• Tests: simulate orphaned architect terminal → verify registry row created on reconcile
• No CLI changes, no HTTP endpoint changes, no schema changes

Related

• afx: removed sibling architect resurrects after Tower recovery — reconciliation trusts persisted row without liveness check #1150 — parent PIR (mirror case: row survives, terminal doesn't). This issue completes the symmetry.
• codev adopt / launchInstance: main architect hijacks an unrelated Claude Code session instead of spawning fresh #1145 — main architect adopt hijack fix (already shipped)
• Tower recovery: Claude architect crash-loops on stale --resume with no user escape #1149 — Claude crash-loop on stale --resume (in flight)
• Subagent recovery-family investigation (2026-07-08) identified the shared design flaw class: "recovery paths trust one signal without cross-verification." This issue is another instance of that flaw the original investigation didn't catch.

Suggested label: area/tower.

[amrmelsayed]

Field datapoint (2026-08-10): an architect added via add-architect ('streamdeck', launched 08-06) vanished across a Tower/machine restart — its architect registry row was gone (permanent-exit cleanup path, presumably), so the workspace-start reconciliation faithfully respawned the five architects that had rows and silently skipped the one that didn't. No error, no report; discovered only when the human noticed the architect 'not responding.' Recovery was clean (add-architect + /arch-init from its state file — the state-file architecture worked exactly as designed), but the silent-skip is this issue's case in miniature: reconciliation should either self-repair a missing row for a known-recent architect or at minimum REPORT the asymmetry ('terminal/state artifacts exist for X but no registration') instead of silently converging on the smaller roster.

[amrmelsayed]

Fresh datapoint (2026-08-12): after today's Tower restart, every builder registration in the codev workspace was gone (afx status listed only a builder spawned post-restart), while their worktrees remain on disk (.builders/pir-1412, pir-1354, spir-1401, pir-1179, plus sibling lanes). Consequence beyond lost visibility: afx cleanup -p <id> fails with "Builder not found for project", so the sanctioned cleanup path is unusable for merged lanes — the only remaining option is manual worktree removal, which policy forbids agents from doing. Registry self-repair (or a cleanup path that reconciles from disk) would close this gap. Sibling architect registrations were dropped too (only main survived via re-init).

[amrmelsayed]

Streamdeck-lane addendum to the 2026-08-12 datapoint, for whoever executes the orphaned-worktree ruling: .builders/pir-1400 hosts the live sideloaded Stream Deck plugin — ~/Library/Application Support/com.elgato.StreamDeck/Plugins/com.cluesmith.codev.sdPlugin is a symlink into that worktree (verified today). Removing it before re-linking breaks the running deck.

Safe order: build apps/streamdeck on the main checkout, re-run streamdeck link against it, then remove the worktree.

For the record, the streamdeck-lane residue is all merged spawn scaffolding and safe to remove once the ruling allows it (pir-1400 only after the re-link above): air-1383, pir-1347, air-1394, air-1388, air-1403, pir-1404, pir-1400.

[amrmelsayed]

Third datapoint (2026-08-12 ~23:00): builder bugfix-1433 — spawned the same evening (~21:44Z), so post-restart with a healthy registration — had its registry row vanish without any Tower restart in the interim (Tower had been up continuously since the morning restart per this session's observations). Discovered when afx cleanup couldn't find it after the lane merged; its worktree was intact and the owner removed it manually on the human's direct word after standard pre-checks. Significance: the two morning datapoints suggested registration loss was restart-scoped; this one shows rows can also disappear during normal operation (possibly on terminal exit/deregistration racing the cleanup). Self-repair should reconcile from disk state, not assume rows only die at restart.

[amrmelsayed]

Fourth datapoint (2026-08-13 ~00:45), confirming the sub-case hypothesized in the previous comment: builder bugfix-1437 — spawned and completed the same night, healthy registration throughout its lane — had its registry row gone by cleanup time, same shape as bugfix-1433 two hours earlier. The pattern is now reproducible: post-completion terminal deregistration orphans the registry row before afx cleanup can use it, forcing manual worktree removal every time. This is a distinct sub-case from the restart wipe (the morning's datapoints) and probably the easier one to fix: cleanup should fall back to resolving the worktree/branch from disk + porch state when the row is missing, rather than hard-failing on 'Builder not found'.

[amrmelsayed]

Correction — strike two of my datapoints. The lane owner re-tested and reports that comments 5273823344 (bugfix-1433) and 5274561144 (bugfix-1437) were operator error, not registry gaps: both were attempted with afx cleanup -i <issue>, which reported not-found, while afx cleanup -p <n> finds and cleans the same builders through the sanctioned path (since verified — rows existed the whole time, both now deregistered properly).

I verified the mechanism in cleanup.ts rather than just accepting the retraction, and it is a real CLI defect rather than mere discoverability: the -i branch matches only b.id === \bugfix-${issue}`plus ab.issueNumberfallback, but actual builder ids carry thebuilder- prefix (builder-bugfix-1433), so the exact match always misses. The -p branch survives the same mismatch **only because it has an extra name-pattern fallback** (b.name.includes(projectId)) that -i` lacks. Filed separately as a bug.

What still stands on this issue: the post-Tower-restart wipe (comment 5264008797), which I hit directly with -p — the fallback-carrying selector — and which left every pre-restart builder unfindable. So #1176 remains a real registry-integrity issue, but the evidence is one restart-scoped wipe, not a general post-completion orphaning pattern. My 'now reproducible' framing in 5274561144 is withdrawn.