Skip to content

fix(relay): mint PTY ids that carry the relay incarnation instead of a restarting counter - #16901

Merged
brennanb2025 merged 9 commits into
mainfrom
brennanb2025/sta5698-pty-id-identity
Aug 30, 2026
Merged

brennanb2025 merged 9 commits into
mainfrom
brennanb2025/sta5698-pty-id-identity

Conversation

@brennanb2025

@brennanb2025 brennanb2025 commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 12 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​269 $\color{#cf222e}{\Huge{\mathbf{−}}}$​138 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​131
Prod 2 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​23 $\color{#cf222e}{\Huge{\mathbf{−}}}$​5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​18

In plain terms

Every terminal on a remote machine gets a name from the helper program that manages it — and the names were just terminal-1, terminal-2, terminal-3, counting up.

The problem: the counter restarted from 1 every time the helper restarted. So terminal-2 today and terminal-2 yesterday could be two completely different terminals. Orca remembers those names to reconnect you to your work, so after a helper restart it could ask for terminal-2 and get something that had nothing to do with what you left open.

Orca papered over this with a pile of guesswork — extra checks trying to work out whether a name still meant what it used to, and a fallback that quietly assumed "probably fine" when it couldn't tell.

Now each name includes a unique tag for that particular helper lifetime. A name from an old helper can never be mistaken for a new one, because a new helper can't produce that name at all.

What you'll notice: nothing directly. This removes a whole class of "Orca reconnected me to the wrong thing" confusion at the root, rather than guarding against it after the fact.

How the three PRs fit together

The original report (STA-5698) was: after the remote helper restarts, Orca might start your AI agent a second time while the first one is still running — two agents editing the same files at once.

That needs two ingredients:

  1. Leftover programs that survive the helper's death — otherwise there's no first agent left to collide with.
  2. Orca being unable to tell whether a remembered terminal name still means the same terminal — otherwise it wouldn't get confused in the first place.

These three PRs remove both ingredients rather than trying to detect the collision afterwards:

Removes Covers
#16894 ingredient 1 the helper crashing on an error
#16895 ingredient 1 the helper being force-killed, running out of memory, or losing power — Windows only, where the OS doesn't clean up for us
#16901 ingredient 2 terminal names that restart from 1 and get reused

Each stands alone and they can land in any order.

Why not just detect the collision? That was the first attempt (#16843, now closed). It tried to work out "has the helper been replaced?" by comparing how long it had been running against when Orca last saw the terminal. It couldn't tell a helper restart apart from the whole machine having rebooted — and after a reboot nothing survives, so there's nothing to protect against. It would have refused to restore your agent in a case that was completely safe. A live reproduction also failed to produce the collision at all on Linux: killing the helper took the agent down with it in 0.3 seconds.

So: prevent the leftovers, fix the names, and the problem doesn't arise instead of being guessed at.


The defect

src/relay/pty-handler.ts mints PTY ids from a per-process counter:

private nextId = 1
id = `pty-${this.nextId++}`

That records a position, not an identity. When a relay restarts with nothing to revive, it starts again at pty-1 — so a client's durable binding to pty-2 can name a different shell in a later relay lifetime.

Scope, precisely: the mint is a do/while that skips ids already taken, and the revive path advances the counter past whatever it parsed. So ids restart only when the new relay revives nothing — which is exactly the case a client reconnecting to a replaced relay hits.

Several things in the tree exist only to compensate for this:

  • attachIdentityMismatches, whose own comment says it "Rejects cross-relay-generation id collisions (a reset relay reuses pty-N)" — and which is permissive when identity is absent
  • expectedPaneKey / expectedTabId on every attach
  • SSH_PTY_IDENTITY_MISMATCH
  • the ambiguity of "not found" from a renumbered namespace, which fix(ssh): a relay that reports a PTY absent must authorise a fresh spawn #16763 has to work around

The change

Mint ids that carry the relay incarnation, following the convention #16691 just established for the same problem class one layer up (src/shared/worktree/identity.ts, wt2:${executionHostId}:${instanceId}):

pty2:<mint-epoch>:<local-sequence>
  • an opaque per-incarnation epoch, not a position
  • a versioned scheme prefix so old and new shapes coexist
  • no mutable locator in the identity

An id minted by a dead incarnation can no longer be minted by a live one.

Backward compatible by construction. The only structural parse of a PTY id anywhere outside tests is the revive path, and it stays legacy-only: old pty-N ids still revive and still advance the counter exactly as before, while new-shaped ids revive opaquely without importing a foreign epoch's sequence. Ids cross the wire as opaque strings — parseAppSshPtyId splits on @@ and takes the remainder, so the added colons are safe — and the one place an id reaches the filesystem hashes it first.

Deliberately not in scope

No client-side behaviour changes, no resume guard, no change to the SSH absence fallback. This is identity hygiene only.

Tests

New coverage proves cross-lifetime uniqueness, legacy revival and counter advancement, foreign new-shape revival, and same-epoch collision freedom. The cross-lifetime assertion was confirmed red before the change (expected pty2, received pty-1).

Sibling suites that asserted the old minted shape were updated to treat the id as opaque rather than to hardcode the new literal — the differential harness now indexes callbacks by the actual spawned id, and the subprocess repair assertions require an id string without coupling to any mint format. Hardcoding pty2:… would have recreated the same coupling one format later.

Verification

pnpm test src/relay/ (the whole directory, not a single file — a single-file run is what hid the sibling breakage in the first place) shows no failures beyond the three that fail identically on origin/main: pty-handler-spawn-environment, pty-handler-startup-command-delivery, and a ~35s timing flake in subprocess. The last of these never calls pty.spawn, so the new mint path is unreachable from it.

pnpm tc, npx oxlint, pnpm oxfmt, and pnpm build:relay all pass, including when merged together with #16894 and #16895.

@coderabbitai

coderabbitai Bot commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

PtyHandler now generates pty2:<mint epoch>:<sequence> IDs and accepts an optional mint epoch. The test harness adds shared ID and handler factory helpers. Spawn, revive, dependency-repair, attach, replay, lifecycle, output, environment, and protocol tests now use generated IDs. New coverage checks sequence behavior, legacy ID revival, foreign mint epochs, epoch encoding, and SSH PTY ID parsing. Output-drain tests register callbacks using IDs returned by spawn.

Merge Risk: 🟠 High · up to fbef6

After a terminal is successfully reattached, a previously scheduled cleanup timer can still force-kill that live terminal several seconds later. This can terminate the user's shell or running work, so the PR is not merge-ready until the timer is canceled or otherwise prevented from reaping restored terminals.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 14 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: PTY IDs now include the relay incarnation instead of relying on a restarting counter.
Description check ✅ Passed The description is detailed, focused, and explains the problem, solution, scope, compatibility, tests, and verification. It mentions issue STA-5698 and explains that there is no visible behavior chang…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Full details: Description check

Explanation

The description is detailed, focused, and explains the problem, solution, scope, compatibility, tests, and verification. It mentions issue STA-5698 and explains that there is no visible behavior change, although it does not reproduce every template heading or checklist item.

  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…tions

The epoch escaping had no test: dropping encodeURIComponent left the whole
relay suite green. Pin the three-field id shape against an epoch that carries
both separators, and cover a colon-bearing relay id through the unchanged
app-side SSH id wrapper.

subprocess.test.ts had traded `pty-1`/`pty-2` for `expect.any(String)`, which
discarded the invariant those two cases exist to prove: an early node-pty load
failure burns no sequence, a late spawn failure burns one.
The harness built the expected id without the encodeURIComponent production
applies at the mint site. A test epoch carrying a reserved character would
diverge silently across ~40 assertions in 11 files.
@brennanb2025

Copy link
Copy Markdown
Contributor Author

Review loop — round 1

Reviewed at head b5c3215645, rebased onto current origin/main before judging. Two test-only commits were pushed during the loop; the production change is unchanged apart from one comment reword.

Verdict

Patched, not superseded. The design is what I would write from scratch: it mirrors the in-repo precedent in src/shared/worktree/identity.ts (wt2:<host>:<instance>), the versioned pty2: prefix lets both shapes coexist, and the epoch is opaque rather than positional. The decision to leave attachIdentityMismatches, the legacy sequence advance, and the SSH absence fallback alone is correct — legacy pty-N ids persist indefinitely through pty.revive, so those guards are still load-bearing for them.

Is this redundant once #16894 and #16895 land?

No — and the relationship runs the other way. Those two prevent orphan processes; this one prevents identifier reuse, which bites with zero orphans present. A client holding a durable binding to pty-2 that reconnects to a replaced relay can be handed a legitimately new, live shell belonging to a different pane. Nothing in #16894 or #16895 touches that.

Three specifics:

  1. nextId only advances from revived legacy ids (pty-handler.ts:2315). fix(relay): reap owned PTYs when the daemon dies on an uncaught exception (STA-5697) #16894 reaps everything on a fatal, so the replacement relay revives nothing — which is exactly the case where the counter restarts at pty-1. Better host-side reaping makes the collision more reliable, not less.
  2. The only guard is permissive by construction: attachIdentityMismatches "only compares fields present on both sides; absent identity stays permissive." A pane without paneKey/tabId still cross-wires.
  3. The client cannot fall back on incarnationId. It lives in incarnationByRelayPtyId on SshPtyProviderOutputState — in-memory, first-write-wins, and cleared by dispose(), so it is gone across exactly the relay replacement that creates the hazard. The durable survivor is the persisted PTY id.

This is also the only one of the three that helps on Linux, where a kill -9 on the relay takes the shell and agent down via SIGHUP in ~0.3 s and there is no orphan to prevent.

Live A/B on a real remote Linux host

Built this branch's relay and the currently shipped relay, ran both against the real node-pty binding on a remote Ubuntu host, two relay lifetimes each, real PTYs:

relay lifetime 1 lifetime 2
shipped pty-1, pty-2 pty-1, pty-2
this branch pty2:8bfb745e-…:1, :2 pty2:ddaf83c0-…:1, :2

The shipped relay reproduces the collision verbatim — identical id strings naming different shells. This branch cannot.

Defects found and fixed

  1. subprocess.test.ts had been made vacuous. The PR replaced { id: 'pty-1' } / { id: 'pty-2' } with expect.any(String) in the only test that drives the real relay subprocess over stdio. Proof: with the production hunk fully reverted, that file stayed green. It also erased a real behavioural signal — the mint happens after loadPty(), so a module that fails to load burns no sequence (repair lands on :1) while a module that loads and whose spawn() throws does (repair lands on :2). Restored as stringMatching(/^pty2:[^:]+:1$/) and /^pty2:[^:]+:2$/ — shape and sequence pinned, epoch left opaque.
  2. encodeURIComponent on the epoch was unpinned. Added a test that constructs a handler with epoch a:b@@c:9 and asserts the minted id still splits into exactly three fields and contains no @@. Removing only the encode now fails exactly that test.
  3. No mixed-version coverage. src/shared/ssh-pty-id.ts is an unchanged old reader that splits an app id on the first @@; nothing pinned that a colon-bearing pty2: id survives it. Added a round-trip test through toAppSshPtyId / parseAppSshPtyId / toRelaySshPtyId, including the wrong-connection rejection.
  4. testPtyId() was an unfaithful double — it built the expectation without the encodeURIComponent production applies, so a test epoch with a reserved character would have diverged silently across ~40 assertions. Now mirrors production.

Ablation

production hunk pinned by
mint line (pty2: + epoch + sequence) 14 tests RED with production reverted, incl. both subprocess.test.ts cases
encodeURIComponent on the epoch exactly 1 test RED when only the encode is removed
constructor field + parameter not independently ablatable — the mint line does not compile without it
two comment rewords not pinnable by a test; comment-only

src/shared/ssh-pty-id.test.ts passes against the pre-fix tree by design: it pins an unchanged module's behaviour against a new-shape id, so it cannot fail pre-fix.

Compatibility

Rule 3 of docs/reference/remote-wire-compatibility.md — the host changes the content of a field old clients read, with no codec change. Safe here: pty-handler.ts:2315 is the only structural parse of a relay PTY id anywhere in the tree, persisted schemas are unconstrained z.string(), and the semantic shift strictly removes a false-equality case rather than adding a claim an old client cannot interpret. Old relay → new client keeps minting pty-N and nothing rejects it; new relay → old client carries pty2: opaquely. Revive re-adopts under the original id in both directions, so the two shapes coexist without collision and no session is lost.

Blast radius is SSH-remote only — PtyHandler is constructed solely by RelayRuntimeServices → relay-daemon.ts. Local terminals use a separate scheme in src/main/daemon/.

Gates

pnpm tc exit 0 · npx oxlint src/relay src/shared clean · git diff --check clean · npx oxfmt no-op · no node:child_process import added, no allowlist or baseline touched.

pnpm test src/relay/ src/shared/ssh-pty-id.test.ts → 1508 passed, 2 failed. Both failures (pty-handler-spawn-environment.test.ts:500, pty-handler-startup-command-delivery.test.ts:167) reproduce identically on a clean origin/main worktree at 2 failed / 46 passed — pre-existing and macOS-only; CI is green on Linux.

Not verified

  • No Windows host validation.
  • No Electron UI validation: this change is unreachable from a local Orca session, so dev-mode CDP would prove nothing about it. The remote-host A/B above is the substitute.
  • The ~35 s subprocess timing flake the PR body mentions did not fire in any of my runs, so I neither confirmed nor cleared it.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

CI follow-up — the one red check is not this PR, but it will block the merge

Head is now 33455d90dc (branch updated onto current main). Everything is green except e2e / ssh docker watcher isolation, which fails on a single spec, reproducibly across two runs at two different bases:

✘ 13 tests/e2e/ssh-pi-compatible-agent-title.spec.ts:147:7
     › classifies OMP and Pi title transitions from a remote terminal
  Error: page.evaluate: Error: No remote worktree found for /tmp/orca-docker-relay-perf-repo
      at connectDockerRemote (tests/e2e/ssh-pi-compatible-agent-title.spec.ts:40:21)
  1 failed, 17 passed (10.9m)

Why this is not a regression from this change

  • The spec is not in this PR's diff.
  • The failure is at repos.addRemote → fetchWorktrees during connection setup, before any terminal is spawned. No PTY id exists at that point in the run.
  • 17 sibling SSH specs in the same job pass, including reattach and restore paths that exercise the new id shape end to end. The job log shows ssh:ssh-…@@pty2:6f886a9c-…:1 routing correctly, and a genuine miss being reported as PTY "pty2:…:1" not found rather than coerced into an exit.
  • The spec carries a stale inline copy of the connect logic. It calls fetchWorktrees(result.repo.id) with no authority arguments and then takes worktreesByRepo[...][0]. The shared helper tests/e2e/helpers/docker-ssh-relay-connection.ts was upgraded in feat: bound direct SSH reconnect fan-out and recovery #11003 to pass { executionHostId, directSshAuthority, requireAuthoritative: true } and to select the worktree by hostId. The duplicate never picked that up. tests/e2e/ssh-localhost.spec.ts:219 has the same stale shape.

Why it appeared only now

The earlier green run of this job at 7eda0e08e6 ran 5 tests in 1.6 min and never executed this spec. The current runs execute 18 tests in 10.9 min. Under #16746 ("trigger the Docker-SSH lane from SSH source and claim every gated spec"), the review commit that touched src/shared/ssh-pty-id.test.ts expanded the lane to every gated spec — which surfaced an already-broken one.

What this means for merging

The lane will be red on this PR, and on any future PR that touches SSH source, until that spec is fixed. The fix is to drop the inline duplicate and call the shared connectDockerRemote helper. I have not pushed that: it is a different subsystem from this PR's stated scope, and I have no Docker SSH rig here to verify an e2e change before pushing it — pushing it blind would be worse than flagging it.

Stated plainly so it is not mistaken for verified: I diagnosed this from the diff, the job logs, and the two connect implementations. I did not run the spec against origin/main, so "pre-existing" is a mechanism-based conclusion rather than a direct A/B.

@brennanb2025

Copy link
Copy Markdown
Contributor Author

Correction — the two local test failures are environmental, not platform-specific

In my first comment I called pty-handler-spawn-environment.test.ts and pty-handler-startup-command-delivery.test.ts "pre-existing and macOS-only". The pre-existing half is right and independently confirmed against origin/main (git checkout origin/main -- src/relay/, same two assertions, 2 failed / 46 passed). "macOS-only" was wrong, and the real cause is more useful:

Both fail because the shell running the suite is itself an Orca agent pane, so ambient ORCA_SEQUENCED_STARTUP_COMMAND / ORCA_SEQUENCED_STARTUP_SCRIPT leak into resolveSetupAgentSequenceLaunchCommand (src/shared/setup-agent-sequencing.ts:20-26) and flip the git-credential guard and the shell-ready feature set.

$ env -u ORCA_SEQUENCED_STARTUP_COMMAND -u ORCA_SEQUENCED_STARTUP_SCRIPT \
    pnpm test src/relay/pty-handler-spawn-environment.test.ts \
              src/relay/pty-handler-startup-command-delivery.test.ts
  Test Files  2 passed (2)
       Tests  48 passed (48)

So they are not a defect on main either — they are a harness-hygiene issue that will hit anyone running the relay suite from inside an Orca terminal. Unrelated to this PR either way; correcting it so nobody chases a phantom platform bug.

Two further notes from the review, neither merge-blocking and neither acted on:

  • Optional consistency nit. The mint encodes the epoch but not the sequence, whereas the wt2: precedent it cites (src/shared/worktree/identity.ts:20-21) encodes both. There the components are free-form and encoding is load-bearing; here randomUUID() is [0-9a-f-]{36} and the sequence is digits, so both encodes are no-ops and the scheme stays injective regardless. Purely a "half-mirror may mislead the next editor" point — worth either encoding both or neither, but not worth churning a merge-ready PR.
  • Pre-existing, hostile-input only. The mint do/while at pty-handler.ts:1587-1590 cannot terminate if nextId is driven to ~1e21 by a revive payload containing pty-<21 digits>, because nextId++ stops advancing at that magnitude. Identical on origin/main; needs a corrupt or hostile revive state to reach.

@greptile-apps

greptile-apps Bot commented Aug 28, 2026 •

Copy link
Copy Markdown

Greptile Summary

The PR namespaces newly minted relay PTY IDs by relay-handler incarnation while preserving legacy and foreign-epoch IDs during revival.

  • Changes fresh IDs from pty-N to pty2:<encoded-epoch>:<sequence>.
  • Keeps legacy revival sequence advancement and treats new-format revived IDs opaquely.
  • Updates relay tests and shared SSH-ID coverage to use opaque, colon-safe identifiers.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
src/relay/pty-handler.ts Introduces an encoded, per-handler mint epoch for fresh PTY IDs while retaining collision checks and legacy revival behavior.
src/relay/pty-handler-test-harness.ts Provides deterministic epoch-scoped IDs and handler construction for tests.
src/relay/pty-handler-spawn-admission.test.ts Covers cross-lifetime uniqueness, separator escaping, legacy revival, and foreign-epoch revival.
src/shared/ssh-pty-id.test.ts Confirms the new colon-bearing relay IDs survive SSH wrapping and routing unchanged.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Relay handler starts] --> B[Generate random mint epoch]
  B --> C[Spawn PTY]
  C --> D["Mint pty2:&lt;epoch&gt;:&lt;sequence&gt;"]
  D --> E[Persist and route ID as opaque string]
  F[Relay replacement] --> G[Generate different epoch]
  G --> H[New PTY cannot reuse old incarnation ID]
  I[Revive serialized PTY] --> J{Legacy pty-N?}
  J -->|Yes| K[Preserve ID and advance legacy sequence]
  J -->|No| L[Preserve foreign ID opaquely]
Loading

Reviews (2): Last reviewed commit: "Merge branch 'main' into brennanb2025/st..." | Re-trigger Greptile

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/relay/pty-handler.ts (1)

1839-1845: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Cancel the pending reap sweep after a successful reattach.

shutdown() arms managed.reapTimer before issuing the kill. If the PTY survives and the client reattaches before the 6-second sweep, this block restores the pane surface but leaves the timer active. The sweep then sees a live PID and calls requestForceKill, terminating the reattached PTY.

Clear managed.reapTimer when the attach is accepted, or make the sweep stop when the pane surface is restored.

Proposed fix
     if (attachedPaneKey) {
       this.retiredPaneSurfaces.restore(attachedPaneKey)
+      if (managed.reapTimer) {
+        clearTimeout(managed.reapTimer)
+        managed.reapTimer = undefined
+      }
     }

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c10e2cb8-6525-4cb1-8757-5aadd39bf027

📥 Commits

Reviewing files that changed from the base of the PR and between b5c3215 and fbef641.

📒 Files selected for processing (1)
  • src/relay/pty-handler.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 4 remain after this review.

@brennanb2025
brennanb2025 merged commit f23d0b1 into main Aug 30, 2026
29 checks passed
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…a restarting counter (stablyai#16901)

* fix(relay): scope PTY ids to mint epochs

* test(relay): treat minted PTY ids as opaque

* test(relay): pin mint-epoch id shape and restore spawn-sequence assertions

The epoch escaping had no test: dropping encodeURIComponent left the whole
relay suite green. Pin the three-field id shape against an epoch that carries
both separators, and cover a colon-bearing relay id through the unchanged
app-side SSH id wrapper.

subprocess.test.ts had traded `pty-1`/`pty-2` for `expect.any(String)`, which
discarded the invariant those two cases exist to prove: an early node-pty load
failure burns no sequence, a late spawn failure burns one.

* test(relay): mirror production epoch escaping in testPtyId

The harness built the expected id without the encodeURIComponent production
applies at the mint site. A test epoch carrying a reserved character would
diverge silently across ~40 assertions in 11 files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant