Repository navigation
Description
Operating system
Linux
Orca version
1.4.237 (orcad 0.1.0+033018f7660d, Electron 43.7.8, Node 24.21.0)
Details
Short summary
On a headless orca-ide serve host, the orcad server stopped responding about 10 seconds after a terminal session was closed from a paired desktop client. The process stayed alive, but it stopped accepting connections and stopped all of its periodic work. Only a kill and restart of the server brought it back.
Setup
- Debian 13 (trixie), 20 vCPU, 26 GB RAM, LXC guest on Proxmox VE.
orca-ide serve --pairing-address wss://<host>runs as a systemd user unit. A reverse proxy forwards to orcad on0.0.0.0:6768.- A macOS desktop client is paired. Several terminals with Claude Code agents run in git worktrees.
- The server had been up for 3.5 days.
Timeline (UTC, from the logs)
10:00:58, orcad logs[history] non-final checkpoint already in flight: <worktreeId>::<repo>@@8d7e6d08for the terminal that is closed later.- During the following hours, the process tree in that terminal's PTY scope used a lot of CPU. When the session ended, systemd reported
orca_pty_orca_daemon_<uuid>_1.scope: Consumed 3h 25min 29s CPU time, 377.8M memory peak. 12:04:25, last entry inorcad.trace.ndjson: routinegit.execspans (status, rev-parse, diff…), which run about every 30 s, each under 100 ms.12:04:36.780,daemon.logshows{"event":"session-killed","sessionId":"<worktreeId>::<repo>@@8d7e6d08","immediate":false,"clientId":"…"}. There is no matchingsession-exitedevent.- After that, nothing at all is logged by orcad: no trace spans, no
[ws-transport] heartbeatlines, no daemon events. The listen socket on:6768showedRecv-Q 40: connections were queued by the kernel but never accepted. Any HTTP or WebSocket request timed out. From12:06, an external probe saw the endpoint as down. 12:12:37,systemctl --user kill -s KILL orca-servethen restart. Over its 3.5-day life, the killed server had used 5h 20min of CPU, with a 4.4 GB memory peak. On restart, orcad logged[daemon] Replacing daemon that failed the health check (health=unreachable, liveSessions=unverifiable, graceRetries=2), thenOrca server ready. Everything worked again.
What was ruled out
- Host resources: guest CPU stayed between 6 and 13 %, memory at about 10 GB out of 26 GB, disk I/O normal. No OOM in the kernel log.
- Network and proxy: a second host with the same setup, behind the same proxy, kept answering during the whole incident. The orca-serve unit stayed
active (running). - Configuration: nothing on the host changed around 12:04.
My reading (not verified)
Closing a session whose history checkpoint was stuck "in flight", after hours of heavy output, seems to have blocked orcad's main event loop, maybe a synchronous flush or a wait that never resolves. The session's history directory was removed when it closed, so I could not inspect its size.
How to reproduce (likely, not confirmed)
- On a headless
orca-ide servehost, open a terminal that produces a lot of output for hours (a busy agent, oryes | head -c 50G). - Wait for a
non-final checkpoint already in flightlog line for that session. - Close that terminal from a paired desktop client.
- Check whether orcad still accepts connections (
ss -ltnshowsRecv-Qgrowing on the listen port) and whether its trace log keeps writing.
Would help
- A watchdog or health check that restarts or reports a blocked orcad event loop. The client and the systemd unit both still saw a "running" server here.
- A bounded session teardown: closing one terminal should never freeze the server.
Activity
tux86 commentedon Oct 27, 2026
Log excerpts from the incident. Repository paths, home directory and UUIDs are redacted. Session-id suffixes are kept, so the killed session (
@@8d7e6d08) can be followed.~/.config/orca/logs/daemon.log, every 2026-10-27 entry up to the restart.The terminal daemon (pid 853926) logs
session-killedat 12:04:36.780, and nothing more: nosession-exited, although systemd shows that the PTY scope ended at the same second. The next line comes from the new daemon, after my restart at 12:12:37. Note also that a second daemon, pid 793, from an older protocol version, was still alive and accepted clients after the restart.~/.config/orca/logs/orcad.trace.ndjson, spans from 12:03:50 onward.The git polling runs every ~30 s, each call under 100 ms. The last span starts at 12:04:24. The next one, at 12:04:54, never happens, and no other span is written until the restart at 12:12:37. The listen socket meanwhile showed
Recv-Q 40.systemd user journal (unit and scope names redacted):
nwparker commentedon Oct 27, 2026
Thanks for the detailed report, @tux86. The listen socket's backlog (
Recv-Q 40), with no timers, heartbeats or trace spans, means orcad's main thread itself stopped. The terminal daemon also went silent. That leaves two likely causes: a garbage-collection or near-out-of-memory stall (your 4.4G peak fits), or a system call stuck in uninterruptible sleep. Next time it hangs, before restarting, could you capture:top -H -p <orcad pid>for a few seconds. Is one thread at ~100% or near 0%?ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'NODE_OPTIONS=--report-on-signal, then runkill -USR2 <orcad pid>during the hang and attach the JSON report it writes.tux86 commentedon Oct 27, 2026
Thanks, this is very helpful. I've set up the captures for the next hang:
orca-servewill run withNODE_OPTIONS="--report-on-signal --report-directory=~/.config/orca/logs". I checked that orcad inherits it and that the Electron launcher accepts it. I'll sendkill -USR2during the hang and attach the report.top -H,/proc/<pid>/{status,wchan,stack}, D-state processes) for both orcad and the terminal daemon before restarting.On point 5: I don't record per-process RSS, but node_exporter covers the host (an LXC container with 12.9 GB). Around the hang that started at 12:04:24 UTC, from 11:30 to 12:20:
So the host was not under memory pressure when orcad stopped. The 4.4G figure is
orca-serve.service's peak over its whole lifetime (5h20 of CPU), not necessarily its size at 12:04. That seems to point away from a host-level near-OOM. It doesn't rule out a V8 heap-limit stall inside orcad.I'll post the captures as soon as it hangs again.
fix(daemon): keep the PTY open until an in-flight force-kill sweep fi…
test(daemon): build the destroy-tracking PTY mock without a type asse…
chore(ratchet): gate event-loop-blocking calls in the daemon, PTY and…
fix(pty): make the local graceful-deadline force-kill async and stub …
fix(pty): hold PTY teardown until an in-flight force-kill sweep settles
fix(pty): let disposal and orphan cleanup join the in-flight force-ki…
fix(daemon): SIGKILL live PTY groups before a shutdown deadline exit
fix(daemon): read the PTY process table without blocking on terminal …
chore(ratchet): gate event-loop-blocking calls in the daemon, PTY and…
test(pty): clean up a job that discovery never captured
1 remaining item
test(pty): scope the group-reap test's ps to its own PTY (#34669)
fix(diagnostics): append daemon and trace logs off the event loop
fix(diagnostics): append daemon and trace logs off the event loop
fix(diagnostics): append daemon and trace logs off the event loop (#3…
fix(orcad): keep orcad's terminal models at the daemon's live depth
fix(orcad): keep provider preference when the reseed snapshot times out
fix(runtime): bound every provider snapshot wait in one place
fix(runtime): reseed once after a late provider snapshot, trust the m…
fix(runtime): never trust a late-reseed model that found no snapshot
fix(orcad): keep terminal models at the daemon's live depth and bound…
fix(remote-runtime): detect a runtime that stays connected but stops …
fix(remote-runtime): detect a runtime that stays connected but stops …
fix(remote-runtime): detect a runtime that stays connected but stops …
fix(remote-runtime): detect a paired host that stays connected but st…