Skip to content

SSH relay leaks orphaned PTY sessions until "Maximum number of PTY sessions reached (50)" #9819

Closed as not planned

Description

@TrentFisher6

Symptom

After heavy use of terminals/worktrees on an SSH remote, spawning any new terminal fails with:

Maximum number of PTY sessions reached (50)
If this persists, please file an issue.

Closing terminals and worktrees does not free slots. Only a full relay reset (Settings → SSH → Reset relay) recovers.

Root cause

The cap lives in the relay daemon (src/relay/pty-handler.ts), which survives app restarts — its shutdown grace window is canceled on every reconnect, so in practice it never restarts. A slot is only freed by an explicit pty.shutdown, and nothing ever reconciles the relay's live PTY list against what the app still tracks. Orphans accumulate through at least three paths:

  1. Close while disconnected — kill() in src/main/ipc/pty.ts finds no provider, tombstones the lease locally, and closes the pane, but the remote PTY is never killed. The next reconnect cancels the relay's grace window, so the orphan lives until the relay dies.
  2. Transient shutdown failure — the provider.shutdown() rejection path logs a warning and closes the pane with no retry; the remote PTY lives on.
  3. Worktree teardown — killAllProcessesForWorktree (src/main/runtime/worktree-teardown.ts) sweeps only the local provider and local registry; SSH relay PTYs outside the live renderer graph are untouched.

Since connection flaps make path 1 common (each agent worktree holds several PTYs), a busy setup hits the 50-slot cap within days.

Proposed fix

After reattachKnownPtys on connect/reconnect, list the relay's PTYs and shut down any pane-bound entry (ORCA_PANE_KEY was set at spawn) that the app no longer owns or leases. Bare relay shells (e.g. remote CLI) are left alone, and entries younger than 30s are skipped so an in-flight spawn racing the sweep can never be reaped.

PR incoming.

Activity

  1. Benjamin-Connelly commented on Jul 24, 2026

    @Benjamin-Connelly

    Hit what looks like the same relay PTY leak on a Linux SSH remote, but with a symptom this thread doesn't mention: the orphaned PTYs don't just exhaust the 50-slot cap — they make the relay pin a CPU core at ~100% indefinitely, which is what ultimately manifests as the client UI hanging and new terminals stalling (terminal_safe_fit_retry_exhausted).

    Root cause at the kernel level: the leaked pty-master fds are closed without being removed from the relay's epoll interest list (EPOLL_CTL_DEL / uv_poll_stop is skipped, and the underlying open file description stays alive via a dup / libuv handle). Their slave ends are hung up, so epoll_pwait returns an unmaskable EPOLLHUP on each dead fd on every loop iteration — forever.

    Evidence (race-free)

    strace -c on a spinning relay — 100% of syscalls are the epoll loop, zero forward progress:

    % time     calls    syscall
    55.19      53443    epoll_pwait
    44.80      53444    io_uring_enter
     0.00          1    write
    (~21,000 iterations/sec)
    

    Froze it (kill -STOP) and compared the epoll interest list to the live fd table:

    # /proc/<pid>/fdinfo/17   (the epoll instance)
    tfd: 34 events: 19 ... ino:59 sdev:5     # 0x19 = EPOLLIN|EPOLLERR|EPOLLHUP
    tfd: 37 events: 19 ... ino:59 sdev:5     # ino:59 sdev:5 == /dev/ptmx (pty master)
    tfd: 39 events: 19 ... ino:59 sdev:5
    ...  ~15 tfds, all /dev/ptmx pty masters ...
    
    # /proc/<pid>/fd/34, 37, 39, 41, 42, 59  ->  all CLOSED / do not exist
    

    So ~15 pty-master fds are registered in epoll but already closed — unremovable by fd number, firing EPOLLHUP permanently. relay-watcher.js was alive and healthy (this is not the parcel-watcher 100%-CPU path from #8112).

    Implication for the proposed fix

    The reconcile-orphaned-PTYs-on-reconnect fix here operates at the session layer, but the spin is a close-path bug: even after a session is reconciled/killed, if the master fd is close()d without first stopping the libuv poll on it, the epoll registration (and the spin) persists. Worth ensuring the teardown does uv_poll_stop / fully .destroy() the wrapping socket before closing the master fd. The dispose path currently no-ops kill on non-Windows (if (process.platform !== "win32") pty.kill = () => {}) and relies on pty.destroy(); on Linux with node-pty 1.1.0 that destroy may not be stopping the poll before close — possibly upstream (cf. node-pty's io_uring interactions, microsoft/node-pty#630).

    Compounding: stale relays across auto-update

    Each app self-update spawns a new relay build-hash dir but leaves the previous --detached --grace-time 0 relay running. Found 3 relay build-hashes each spinning at ~100% for 1–2 days, pushing load to 6.0 on a 6-core host. Reaping prior-build relays on update would bound the blast radius.

    Diagnostic for confirming the spin

    # epoll fd N from `epoll_pwait(N, ...)` in an strace sample, then:
    sudo kill -STOP <relay-pid>
    sudo cat /proc/<relay-pid>/fdinfo/<N>   # tfd entries
    ls -l /proc/<relay-pid>/fd/             # tfd numbers absent here == leaked
    sudo kill -CONT <relay-pid>
    

    Environment: Orca 1.4.152, relay 0.1.0+8d4e15ad63eb, remote Linux kernel 6.8.0, bundled Node v18.19.1, node-pty 1.1.0, SSH Worktree. Related: #9034, #9229.

  2. nwparker commented on Jul 27, 2026

    @nwparker
    Contributor

    Reproduction status: env_blocked (SSH relay PTY session leak)

    Maximum PTY sessions leak needs long-lived SSH relay under load.

    Not labeling cannot_repro.

  3. Benjamin-Connelly commented on Aug 5, 2026

    @Benjamin-Connelly

    Still reproducing on 1.4.171 (released today, after the 1.4.169–1.4.171 SSH/remote-runtime patch series). Relay build 0.1.0+1fc53198b5e1, deployed to the remote host at 12:16 today. Remote host is Ubuntu; client is Linux.

    Posting because the existing data here is from 1.4.153-rc.4, and #12449 ("SSH reattach failure paths never kill abandoned relay PTYs") was closed as completed today — so it's worth recording that the leak survives that fix.

    Reproduction

    1. Connect to a remote SSH host, open ~5 terminals.
    2. Close all terminals.
    3. Inspect the remote host.

    Opening ~5 terminals created 8 shells server-side. After closing every terminal, 4 shells remained, and the relay still held 4 /dev/ptmx masters for them:

    pid=2203252  stat=Ss+  tty=pts/5  wchan=do_select  children=[none]
    pid=2203613  stat=Ss+  tty=pts/6  wchan=do_select  children=[none]
    pid=2203946  stat=Ss+  tty=pts/8  wchan=do_select  children=[none]
    pid=2204255  stat=Ss   tty=pts/9  wchan=do_wait    children=[claude]
    

    Three are idle bash at a prompt with no children — nothing running, no reason to persist. The fourth legitimately holds a foreground process.

    The other four shells (pts/0,1,3,4) were reaped correctly. That asymmetry seems like the useful signal: if session persistence were intended, all eight would survive. Half did.

    The relay spins with zero clients attached

    CPU climbed monotonically after every terminal was closed and nothing was connected:

    1.8%  -> 10.4% -> 11.1% -> 11.7% -> 12.3% -> 12.9%  (over ~1 min)
          -> 35.2%                                       (a few min later)
    

    13.5% average over its lifetime — 1m18s of CPU in 9m36s of wall clock, rising.

    On the previous build (55049cb9647d) the same process reached 99.9% and held it for 4 days 15 hours, which is how this was noticed at all: the symptom presented as "SSH to that host feels laggy", not as an editor fault. Host-wide iowait was 25% and dropped to 0% when the relay was killed.

    Two details that may matter for the fix

    1. SIGTERM does not kill the leaked shells. All three empty ones ignored SIGTERM and required SIGKILL.

    2. Reaping the leaked PTYs does not stop the spin. After killing 3 of the 4 leaked shells, the relay correctly released their masters — ptmx went 4 → 1 — but CPU continued climbing:

    after kill:  35.8% -> 37.4% -> 37.7% -> 38.0% -> 38.3% -> 38.6%   (ptmx=1 throughout)
    

    So the runaway isn't a function of how many PTYs are currently held. Once it starts, releasing the descriptors doesn't recover it — the relay process itself has to be killed. That means "close the leaked terminals" isn't a viable workaround; the relay must be terminated and restarted.

    That distinction seems relevant to #9820: reaping orphaned PTYs on reconnect would stop the session cap from being exhausted, but on this evidence it would not stop an already-spinning relay from burning a core.

    Environment

    • orca-ide 1.4.171 (deb), Linux client
    • relay 0.1.0+1fc53198b5e1
    • remote: Ubuntu, connected over a WireGuard-based mesh, ControlPersist 8h
    • previously observed on 0.1.0+55049cb9647d and 0.1.0+8d4e15ad63eb
  4. ai-daming commented on Aug 6, 2026

    @ai-daming

    Complementary repro — macOS client → Linux remote, relay build
    0.1.0+1fc53198b5e1 (same build @Benjamin-Connelly reported earlier today).
    Posting because the macOS side gives a one-step repro trigger, and my logs
    show a slightly cleaner failure mode worth recording.

    Environment

    • Client: macOS, orca 1.4.169
    • Remote: Ubuntu 26.04 LTS, kernel 7.0.0-28-generic, x86_64, relay build
      0.1.0+1fc53198b5e1 (--detached --grace-time 0)
    • @Benjamin-Connelly already confirmed the leak survives on the newer 1.4.171
      (released today, after the 1.4.169–1.4.171 SSH/remote-runtime patch series);
      I see the same leak on 1.4.169, and the leaking relay build is identical,
      so this spans the whole patch series and is independent of the client minor.

    Topology / why macOS matters here

    • The remote runs the orca relay — the leak is entirely server-side, consistent
      with the Linux-client repros above.
    • The macOS-specific part is the trigger: closing the laptop lid = an abrupt
      network drop with no graceful close. That hits path 1 (close while
      disconnected)
      on every sleep — reliably and repeatedly, no manual disconnect
      needed. A single outing leaked a batch of PTYs; the cap was hit within a day
      of normal mobile use.

    Repro

    1. macOS client, connect to a Linux remote via orca SSH remoting, open a handful
      of terminals / worktrees.
    2. Close the Mac lid (sleep) → wake → reconnect. Repeat 2–3×.
    3. New terminals fail with Maximum number of PTY sessions reached (50).

    Evidence — ptys only ever goes up, never down

    From relay.log over a single day with two outings:

    after disconnect (UTC) relay ptys=
    03:38 13
    04:48 23
    04:52 27
    04:55 29
    07:47 (after ~3h lid-sleep gap) 42 (+13 across one outing)
    10:48 44 → 46
    10:51 50 (cap hit)

    Across that window there were 24 Grace started events and 0 PTYs ever
    reaped
    — ptys is strictly monotonic. Notably Grace canceled did not
    fire today, so in this config the reaper isn't even being canceled by
    reconnects; with --grace-time 0 it simply reaps nothing. That matches the
    OP's "a slot is only freed by an explicit pty.shutdown," but it's worth
    flagging that --grace-time 0 leaves no reaping safety net at all — the
    cap is reached purely from accumulation, with zero recovery between flaps.

    Two other notes

    • Hit the 50-cap variant without any CPU spin: relay sat at ~1.2%, load
      avg ~0.07. So the session-layer leak and the epoll-spin are independent —
      you can exhaust the cap with a perfectly idle relay.
    • Also had 3 stale relay build-hash dirs left around by auto-updates (only
      one in use) — same compounding @nwparker described.

    Workaround that cleared it

    kill -9 the relay.js process(es); orca respawns a clean one on next
    connect. The leaked shells ignored SIGTERM and required SIGKILL, same as
    above.

  5. Benjamin-Connelly commented on Aug 9, 2026

    @Benjamin-Connelly

    Two further data points from the same Linux host, relay build 0.1.0+66c426c5173c, today.

    The spin is unbounded, and the log shows zero forward progress

    A --detached --grace-time 0 relay ran for 1d14h and accumulated 38h32m of CPU time — sustained 100-107% of one core on a 6-core host. Over that entire window relay.log contains nothing but paired stream records:

    [relay] stream start id=1872 size=507
    [relay] stream end id=1872
    

    No errors, no reconnects, no EBADF. From the log alone the relay looks idle and healthy, which makes this expensive to notice: the only symptom the user sees is that new terminals stop opening.

    The relay's own CLI fails while the host still reports connected

    While spinning, orca config list over the relay socket returns:

    [relay-connect] Handshake OK at version=0.1.0+66c426c5173c
    [orca-cli] Relay socket error: write EPIPE
    

    The handshake succeeds and only the subsequent write fails. This is a useful liveness probe — it distinguishes a wedged relay from a slow one without needing /proc access, and it means the client's "connected" indicator is not derived from anything that survives the wedge.

    Why the failure surfaces as "cannot open terminals"

    Worth stating explicitly, since it is a second ceiling independent of the 50-PTY cap: the relay multiplexes every terminal as a channel on a single SSH connection, so each terminal counts against sshd's MaxSessions, which defaults to 10. Seven leaked shells is enough to exhaust it, and new terminals then fail at the transport layer before any relay logic runs. Hosts that raise MaxSessions will hit the 50-PTY cap from this issue's title instead; hosts that do not will hit sshd first and see a less obvious error.

    Distinguishing leaked shells from live ones, without guessing

    For anyone writing a watchdog: "childless shell" is not a safe signal, because a terminal the user has open and idle is indistinguishable from a leaked one by that test — reaping on it closes live tabs. The relay holds each live PTY's master as /dev/ptmx, and /proc/<relay>/fdinfo/<fd> reports the slave number as tty-index, so the live set is exactly knowable:

    for fd in /proc/$RELAY/fd/*; do
      [ "$(readlink "$fd")" = /dev/ptmx ] || continue
      awk '/^tty-index:/{print $2}' /proc/$RELAY/fdinfo/"${fd##*/}"
    done

    Any relay-spawned shell on a pts/N outside that set is leaked. Reaping those does not stop the spin, which is consistent with the epoll analysis above: the fds driving it are already closed, so nothing addressable by pid or fd number remains. Killing the relay is the only thing that clears it.

  6. marked [Bug]: #16749 as a duplicate of this issue on Sep 27, 2026
  7. pius-hwang commented on Oct 2, 2026

    @pius-hwang

    Still reproducing on 1.4.218, macOS client to macOS remote, relay 0.1.0+f6e4b640b122. The trigger here needs no user action at all: maintenance DarkWakes while the lid stays closed.

    What happened

    The client laptop sat closed overnight (powernap 1). Three DarkWakes each produced a short relay connection (about 20 s), and each one left a fresh batch of zsh -l behind:

    Client DarkWake (pmset -g log, KST) Relay Socket client accepted (KST) Relay ptys= at the next Grace started
    (before) 5
    19:01:32 19:01:37 30 (+25)
    20:02:33 20:02:38 36 (+6)
    20:30:13 20:30:23 50 (+14, cap)
    • Every shell in a batch started within the same two seconds, about 10 s after the accept. The client logged Entering Sleep state due to 'Maintenance Sleep' 5 to 10 s after each wake.
    • The 12 relay connections later that night all ended at ptys=50. In the morning the first new terminal failed with Maximum number of PTY sessions reached (50).
    • 47 of the 50 PTYs were childless login shells sitting at a prompt (Ss+). The other 3 were running claude. All 50 were still counted by the relay, so this is the relay-owned variant, not the closed-master variant described above.
    • These are bare shells with nothing typed into them, so this does not look like the SSH: sleeping-agent wake sweep forks live remote sessions (disconnected panes judged dead; stale worktreeId; per-worktree dedupe) #12699 wake sweep (claude --resume).

    The only client trace spans in those windows are a terminal_park_verdict_churn breadcrumb (trigger: burst, flips: 3, same tab both times) and:

    10:01:40Z persistence.pty-binding {"binding.host":"ssh","binding.origin":"reattach","binding.generation_gap":27,"binding.eligible":false,"binding.misses":"not_durable","binding.outcome":"flushed"}
    11:30:25Z persistence.pty-binding {"binding.host":"ssh","binding.origin":"reattach","binding.generation_gap":1,"binding.eligible":false,"binding.misses":"not_durable","binding.outcome":"flushed"}
    

    My reading, not confirmed from the code: the reattach path ran during the DarkWake, found the bindings not durable, and the panes were spawned fresh instead of reattached. The client went back to sleep before anything could reconcile.

    Why the batches never expire

    The relay was started with --grace-time 0, and prepareForHostSleep() also sends graceTimeSeconds: 0 before the host sleeps. The relay treats timeoutMs === 0 as no deadline. Every disconnect that night logged Grace started (socket client closed): timeoutMs=0, branch=configured and no Grace expired. So a batch created during a DarkWake has no expiry at all.

    Notes for anyone working around it on macOS

    • Finding relays: ps aux | grep relay | grep -v grep misses the newer relays, because their command line includes --ripgrep-path. Use pgrep -fl 'node relay[.]js' instead. Doing that here found 14 --detached relays from 10 builds across two clients, holding about 147 PTYs between them. The superseded builds are the [Bug]: Direct SSH: updating Orca strands all remote terminals and running agent sessions (relay version swap) #13852 / [Bug]: Updating Orca strands SSH relays and duplicates remote agents #21215 problem.
    • Stopping idle shells without killing the relay: kill -HUP stops the idle shells cleanly (they ignore SIGTERM, as noted above). The relay drops them from its map, and new terminals open again right away. A superseded relay left with 0 PTYs exits by itself 15 min later (branch=idle-no-ptys). Unlike killing the relay, this keeps running claude sessions alive.
    • Caveat: on a macOS remote there is no /proc/<pid>/fdinfo, so I could not tell leaked shells from open but idle tabs. I only reaped shells whose tty had seen no input for hours (tty atime).

    Environment: client macOS 27.0.1, Orca 1.4.218. Remote macOS 26.5.2 arm64, relay 0.1.0+f6e4b640b122 on Homebrew node.

  8. OrcaWin commented on Oct 11, 2026

    @OrcaWin
    Collaborator

    Closing as obsolete after the merged SSH relay retirement in #27214 and #27215. This issue concerns the SSH relay PTY lifecycle, recovery, backpressure, or orphan cleanup. #27214 removed the desktop relay connection/fallback path; #27215 deleted the SSH relay server, deployment assets, and relay-only tests. SSH hosts now use the managed Orca server (orcad), and unsupported hosts fail explicitly, so this relay-specific work is no longer planned. Existing relays started by older Orca versions are not automatically stopped by these merges; their terminals must be closed in that older version or ended on the host before conversion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions