Skip to content

Drain an older terminal host in the background, or restart its terminals now - #374

Merged
benfriebe merged 1 commit into
mainfrom
issue-372-drain-old-host
Oct 11, 2026
Merged

benfriebe merged 1 commit into
mainfrom
issue-372-drain-old-host

Conversation

@benfriebe

Copy link
Copy Markdown
Owner

Closes #372

Problem

A terminal host keeps the code it started with, node-pty included, until it exits. The app never stops the daemon, and an update or kelpi daemon restart hands every terminal to the host already running. So a fix inside the host, like #371's pty leak, only reaches a machine after a full stop, which ends every shell.

Change

Host generations

  • HOST_PROTOCOL_VERSION goes from 1 to 2. It is now the host's generation, bumped whenever a running host has to be replaced.
  • DRAINABLE_HOST_PROTOCOLS lists the older generations a daemon drains rather than ends, each with the reason the user is shown. Generation 1 is there because it can leak a pty for every terminal it opens.
  • host-generation.test.ts fails if node-pty's version changes without HOST_NODE_PTY_VERSION, and its message says to bump the generation.

Draining (boot/compose.ts)

  • On boot the daemon adopts the current host, then each drainable host it finds a pid record for. It reads the old host's token and says hello with the old protocol, the only one a generation 1 host accepts.
  • A draining host keeps its terminals and gets no new ones. When its last terminal ends (TerminalHostClient.onEmpty), the daemon shuts it down.
  • Panes whose shell exited while no daemon ran are closed only after every host has been read, so a pane living on the old host is never closed by mistake.
  • Handoff holds, checkpoints and detaches every host, so a drain carries on across updates and restarts. Stop shuts all of them down. If a draining host dies, its panes are respawned on the current host and their agents resumed, as for the current host.
  • A generation that is neither current nor drainable is still ended at boot.

Restarting old terminals now

restart-old-terminals is a new wire verb (explicit chain, owner-only for paired devices). It:

  1. reads each old shell's real folder with lsof and saves it to the pane;
  2. captures each pane's agent session, and mutes agent hook events from the old shells while they end;
  3. forgets the old terminals, shuts the old host down and waits for it to exit (SIGKILL after 3 s), so whatever it leaked is free before anything new spawns;
  4. starts a fresh shell for each pane on the current host and types claude --resume <id> or codex resume <id>.

Only one runs at a time (a second request joins the first), a handoff or stop waits for it to finish, and it is refused when the daemon has no current host (terminals running in-process).

Telling the user

  • Card: a new WS message, old-terminal-hosts, is sent on change and after every snapshot. It drives a card in the toast corner saying how many terminals and agent sessions are on the old host, why it is being retired, and what restarting costs.
    • "Later" hides it for that host, remembered by pid and start time.
    • The palette row "Restart Old Terminals…" brings it back.
    • "Restart terminals" shows "Restarting…" until the restart is done.
  • CLI: kelpi daemon status (and kelpid status, --json too) lists the current host and each old one. kelpi daemon restart-old-terminals says what it restarts and asks first at a terminal; --force skips the question.
  • Ping: gains a terminal_hosts block.

Host version

Every released host reports its version as 0.3.0-dev: only the daemon gets KELPID_VERSION, so the host falls back to the compiled default. New hosts get KELPID_VERSION passed through. The card and status leave out an old host's version, since it would be wrong.

Docs

  • docs/terminal-host.md: §2 (H = 2), §8 (a draining host dying, forced stop), §10, §11 rewritten (generations, draining, restarting old terminals, telling the user), §12 tests.
  • docs/wire-protocol.md: the new verb (§6.8.1), the allowlist and the ping block.

Testing

  • Hand test in the app (owner): a dev instance on main (generation 1) with several terminals and a Claude session, handed over with SIGUSR2 to a dev instance of this branch on the same state. It worked.
  • Against the real v0.2.15 release:
    • A v0.2.15 daemon and host (node-pty 1.1.0) leaked 10 ptys (ptmx 4 → 14).
    • After a handoff to this branch's daemon, it adopted the terminals from the generation 1 host and drained it.
    • In headless Chrome: the card, Later (still hidden after a reload), the palette row, and Restart terminals. Both panes moved to the current host, a stub Claude resumed --resume <id> in its folder, and the old host exited.
    • kelpid restart-old-terminals did the same for 3 terminals, and a second run said there was nothing to do.
  • New tests:
    • boot/terminal-host.test.ts, with real host processes and a generation 1 host built by rewriting the constant: draining across two handoffs with a new pane going to the current host; a restart into the live folder with the resume typed once; a handoff in the middle of a restart, and two restarts joining; refusing with no current host; retiring at boot; a draining host crashing; an undrainable generation ended.
    • host.test.ts: live counts and onEmpty; a mismatched protocol refused.
    • main-restart-old.test.ts: the prompt, yes and no, --force, scripts, nothing to do, status text and JSON.
    • Also host-generation.test.ts, the agent event mute, ping, the bridge, the palette, the card copy and the allowlist count.
  • pnpm check: root 646 files / 10312 tests and shell 1057 tests passed. One run hit EADDRINUSE in tailnet-binding.test.ts, which passed 3/3 on its own.
  • node scripts/verify.mjs --full: typecheck, root tests, shell tests, bundles, the full audit and the packaged smoke (74/74; ping shows terminal_hosts with generation 2) passed.

Reviewer notes

A fresh Claude session and Codex reviewed the branch. Both flagged a handoff or stop arriving during a restart; both now wait for the restart to finish. Also fixed from the reviews:

  • refuse a restart when the daemon has no current host;
  • save each recovered folder to its pane;
  • capture agent sessions before the old host is ended, on host loss too;
  • report an error when a restart is cut short;
  • make the verb owner-only for paired devices;
  • key "Later" by start time as well as pid;
  • run the lsof lookups in parallel;
  • add the interleaving tests above.

Not covered

  • No redraw after a restart. An open window keeps the old screen, and the new shell's output appears below it, as after a host crash today.
  • The resume command shows twice on screen (the tty echo, then the shell's redraw), but runs once.
  • In-process fallback while a host drains: if the current host could not start, so new terminals run in the daemon, a handoff keeps the draining host's terminals but ends the in-process ones, and still reports "handed-off".

🤖 Generated with Claude Code

…als now (#372)

A terminal host keeps the code it started with, node-pty included, until it exits, and the
app never stops the daemon, so a fix in the host (#370's pty leak) reached a machine only
after a full stop. The host generation is now bumped whenever a running host has to be
replaced, and a daemon drains a host of an older generation it can still speak to instead of
adopting it: that host keeps the terminals it has, gets no new ones, and is shut down when
its last one ends.

The app shows a card while one drains. Later hides it; Restart terminals (or `kelpi daemon
restart-old-terminals`) restarts those panes on the current host: each shell starts again
where its old one was, its Claude or Codex session is resumed, and the old host is waited
out so what it leaked is free before anything new spawns. `kelpi daemon status` lists both
hosts.

Generation 2 starts with this change, so every machine running an older host drains it after
the update. A test now fails if node-pty moves without a new generation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@benfriebe
benfriebe merged commit a222213 into main Oct 11, 2026
@benfriebe
benfriebe deleted the issue-372-drain-old-host branch October 11, 2026 02:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A running terminal host keeps its old code through every update, so a host or node-pty fix needs a full stop to land

1 participant