Skip to content

[Bug]: headless orcad stops accepting connections after a terminal with a stuck history checkpoint is closed #34410

Closed

Description

@tux86

Operating system

Linux

Orca version

1.4.237 (orcad 0.1.0+033018f7660d, Electron 43.7.8, Node 24.21.0)

Details

Short summary

On a headless orca-ide serve host, the orcad server stopped responding about 10 seconds after a terminal session was closed from a paired desktop client. The process stayed alive, but it stopped accepting connections and stopped all of its periodic work. Only a kill and restart of the server brought it back.

Setup

  • Debian 13 (trixie), 20 vCPU, 26 GB RAM, LXC guest on Proxmox VE.
  • orca-ide serve --pairing-address wss://<host> runs as a systemd user unit. A reverse proxy forwards to orcad on 0.0.0.0:6768.
  • A macOS desktop client is paired. Several terminals with Claude Code agents run in git worktrees.
  • The server had been up for 3.5 days.

Timeline (UTC, from the logs)

  1. 10:00:58, orcad logs [history] non-final checkpoint already in flight: <worktreeId>::<repo>@@8d7e6d08 for the terminal that is closed later.
  2. During the following hours, the process tree in that terminal's PTY scope used a lot of CPU. When the session ended, systemd reported orca_pty_orca_daemon_<uuid>_1.scope: Consumed 3h 25min 29s CPU time, 377.8M memory peak.
  3. 12:04:25, last entry in orcad.trace.ndjson: routine git.exec spans (status, rev-parse, diff…), which run about every 30 s, each under 100 ms.
  4. 12:04:36.780, daemon.log shows {"event":"session-killed","sessionId":"<worktreeId>::<repo>@@8d7e6d08","immediate":false,"clientId":"…"}. There is no matching session-exited event.
  5. After that, nothing at all is logged by orcad: no trace spans, no [ws-transport] heartbeat lines, no daemon events. The listen socket on :6768 showed Recv-Q 40: connections were queued by the kernel but never accepted. Any HTTP or WebSocket request timed out. From 12:06, an external probe saw the endpoint as down.
  6. 12:12:37, systemctl --user kill -s KILL orca-serve then restart. Over its 3.5-day life, the killed server had used 5h 20min of CPU, with a 4.4 GB memory peak. On restart, orcad logged [daemon] Replacing daemon that failed the health check (health=unreachable, liveSessions=unverifiable, graceRetries=2), then Orca server ready. Everything worked again.

What was ruled out

  • Host resources: guest CPU stayed between 6 and 13 %, memory at about 10 GB out of 26 GB, disk I/O normal. No OOM in the kernel log.
  • Network and proxy: a second host with the same setup, behind the same proxy, kept answering during the whole incident. The orca-serve unit stayed active (running).
  • Configuration: nothing on the host changed around 12:04.

My reading (not verified)

Closing a session whose history checkpoint was stuck "in flight", after hours of heavy output, seems to have blocked orcad's main event loop, maybe a synchronous flush or a wait that never resolves. The session's history directory was removed when it closed, so I could not inspect its size.

How to reproduce (likely, not confirmed)

  1. On a headless orca-ide serve host, open a terminal that produces a lot of output for hours (a busy agent, or yes | head -c 50G).
  2. Wait for a non-final checkpoint already in flight log line for that session.
  3. Close that terminal from a paired desktop client.
  4. Check whether orcad still accepts connections (ss -ltn shows Recv-Q growing on the listen port) and whether its trace log keeps writing.

Would help

  • A watchdog or health check that restarts or reports a blocked orcad event loop. The client and the systemd unit both still saw a "running" server here.
  • A bounded session teardown: closing one terminal should never freeze the server.

Activity

  1. tux86 commented on Oct 27, 2026

    @tux86
    Author

    Log excerpts from the incident. Repository paths, home directory and UUIDs are redacted. Session-id suffixes are kept, so the killed session (@@8d7e6d08) can be followed.

    ~/.config/orca/logs/daemon.log, every 2026-10-27 entry up to the restart.

    The terminal daemon (pid 853926) logs session-killed at 12:04:36.780, and nothing more: no session-exited, although systemd shows that the PTY scope ended at the same second. The next line comes from the new daemon, after my restart at 12:12:37. Note also that a second daemon, pid 793, from an older protocol version, was still alive and accepted clients after the restart.

    {"src":"daemon","ts":"2026-10-27T08:08:58.998Z","pid":853926,"event":"startup-command-delivery","sessionId":"<uuid>::<repo>@@9b794ca6","written":true,"hasCommand":true,"commandLength":19,"viaShellArgs":false,"queuedByShellReadyBarrier":true}
    {"src":"daemon","ts":"2026-10-27T08:08:58.999Z","pid":2916078,"event":"session-created","sessionId":"<uuid>::<repo>@@9b794ca6"}
    {"src":"daemon","ts":"2026-10-27T08:48:26.784Z","pid":793,"event":"session-killed","sessionId":"serve-<uuid>","immediate":false,"clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T08:48:26.789Z","pid":793,"event":"session-exited","sessionId":"serve-<uuid>","code":1,"cause":"exited"}
    {"src":"daemon","ts":"2026-10-27T10:38:14.915Z","pid":853926,"event":"startup-command-delivery","sessionId":"serve-<uuid>","written":false,"hasCommand":false,"commandLength":0,"viaShellArgs":false,"queuedByShellReadyBarrier":false}
    {"src":"daemon","ts":"2026-10-27T10:38:14.915Z","pid":3007318,"event":"session-created","sessionId":"serve-<uuid>"}
    {"src":"daemon","ts":"2026-10-27T10:38:29.231Z","pid":853926,"event":"startup-command-delivery","sessionId":"<uuid>::<repo>@@KMrKNYnm","written":true,"hasCommand":true,"commandLength":39,"viaShellArgs":false,"queuedByShellReadyBarrier":true}
    {"src":"daemon","ts":"2026-10-27T10:38:29.231Z","pid":3007765,"event":"session-created","sessionId":"<uuid>::<repo>@@KMrKNYnm"}
    {"src":"daemon","ts":"2026-10-27T12:04:36.780Z","pid":853926,"event":"session-killed","sessionId":"<uuid>::<repo>@@8d7e6d08","immediate":false,"clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.843Z","pid":3069497,"event":"startup","protocolVersion":50,"socketPath":"<home>/.config/orca/daemon/daemon-v50.sock"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.868Z","pid":3069497,"event":"terminal-scopes","slice":"orca_ptys_orca_daemon_50a21b4a_1870_4f3a_949a_48f93e86af4b.slice","properties":["OOMPolicy=continue","TimeoutStopSec=5s"]}
    {"src":"daemon","ts":"2026-10-27T12:13:14.909Z","pid":3069497,"event":"ready"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.910Z","pid":3069497,"event":"client-hello-accepted","role":"control","clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.911Z","pid":3069497,"event":"client-hello-accepted","role":"stream","clientId":"<uuid>","streamFraming":"binary-v1"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.913Z","pid":3069497,"event":"client-hello-accepted","role":"control","clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.913Z","pid":3069497,"event":"client-hello-accepted","role":"stream","clientId":"<uuid>","streamFraming":"binary-v1"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.922Z","pid":793,"event":"client-hello-accepted","role":"control","clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T12:13:14.924Z","pid":793,"event":"client-hello-accepted","role":"stream","clientId":"<uuid>"}
    {"src":"daemon","ts":"2026-10-27T12:13:15.060Z","pid":3069497,"event":"client-hello-accepted","role":"control","clientId":"health-check"}
    {"src":"daemon","ts":"2026-10-27T12:13:19.257Z","pid":3069497,"event":"startup-command-delivery","sessionId":"<uuid>::<repo>@@9b794ca6","written":false,"hasCommand":false,"commandLength":0,"viaShellArgs":false,"queuedByShellReadyBarrier":false}
    {"src":"daemon","ts":"2026-10-27T12:13:19.257Z","pid":3069681,"event":"session-created","sessionId":"<uuid>::<repo>@@9b794ca6"}
    {"src":"daemon","ts":"2026-10-27T12:13:19.282Z","pid":28831,"event":"session-attached","sessionId":"<uuid>::<repo>@@9ulwHgmU"}
    {"src":"daemon","ts":"2026-10-27T12:13:27.950Z","pid":3069497,"event":"startup-command-delivery","sessionId":"<uuid>::<repo>@@4jR6v4Y7","written":true,"hasCommand":true,"commandLength":39,"viaShellArgs":false,"queuedByShellReadyBarrier":true}
    {"src":"daemon","ts":"2026-10-27T12:13:27.950Z","pid":3070134,"event":"session-created","sessionId":"<uuid>::<repo>@@4jR6v4Y7"}
    {"src":"daemon","ts":"2026-10-27T12:13:44.917Z","pid":3069497,"event":"process-growth-level","level":"ok"}
    

    ~/.config/orca/logs/orcad.trace.ndjson, spans from 12:03:50 onward.

    The git polling runs every ~30 s, each call under 100 ms. The last span starts at 12:04:24. The next one, at 12:04:54, never happens, and no other span is written until the restart at 12:12:37. The listen socket meanwhile showed Recv-Q 40.

    start     name      duration  subcommand  exit
    12:03:53 git.exec 14 ms status Success
    12:03:54 git.exec 26 ms rev-parse Success
    12:03:54 git.exec 34 ms branch Success
    12:03:54 git.exec 77 ms rev-parse Success
    12:03:54 git.exec 51 ms rev-parse Success
    12:03:54 git.exec 10 ms merge-base Success
    12:03:54 git.exec 22 ms rev-list Success
    12:03:54 git.exec 48 ms diff Success
    12:04:24 git.exec 24 ms rev-parse Success
    12:04:24 git.exec 33 ms branch Success
    12:04:24 git.exec 33 ms rev-parse Success
    12:04:24 git.exec 8 ms rev-parse Success
    12:04:24 git.exec 9 ms merge-base Success
    12:04:24 git.exec 40 ms diff Success
    12:04:24 git.exec 40 ms rev-list Success
    (no span until the restart at 12:12:37)
    

    systemd user journal (unit and scope names redacted):

    12:04:36 systemd[153]: orca_pty_orca_daemon_<uuid>_1.scope: Consumed 3h 25min 29.362s CPU time, 377.8M memory peak.
    12:12:37 systemd[153]: orca-serve.service: Consumed 5h 20min 23.162s CPU time, 4.4G memory peak, 166.2M memory swap peak.
    12:13:11 orca-ide: [daemon] Replacing daemon that failed the health check (health=unreachable, liveSessions=unverifiable, graceRetries=2)
    
  2. nwparker commented on Oct 27, 2026

    @nwparker
    Contributor

    Thanks for the detailed report, @tux86. The listen socket's backlog (Recv-Q 40), with no timers, heartbeats or trace spans, means orcad's main thread itself stopped. The terminal daemon also went silent. That leaves two likely causes: a garbage-collection or near-out-of-memory stall (your 4.4G peak fits), or a system call stuck in uninterruptible sleep. Next time it hangs, before restarting, could you capture:

    1. top -H -p <orcad pid> for a few seconds. Is one thread at ~100% or near 0%?
    2. For both orcad and the terminal daemon:
      grep -E 'State|VmRSS|VmHWM' /proc/<pid>/status
      cat /proc/<pid>/wchan
      sudo cat /proc/<pid>/stack
      
    3. ps -eo pid,stat,wchan:32,cmd | awk '$2 ~ /D/'
    4. If you can, start serve with NODE_OPTIONS=--report-on-signal, then run kill -USR2 <orcad pid> during the hang and attach the JSON report it writes.
    5. Any RSS trend for orcad over its uptime, if you have monitoring.
  3. tux86 commented on Oct 27, 2026

    @tux86
    Author

    Thanks, this is very helpful. I've set up the captures for the next hang:

    • orca-serve will run with NODE_OPTIONS="--report-on-signal --report-directory=~/.config/orca/logs". I checked that orcad inherits it and that the Electron launcher accepts it. I'll send kill -USR2 during the hang and attach the report.
    • I'll capture steps 1 to 3 (top -H, /proc/<pid>/{status,wchan,stack}, D-state processes) for both orcad and the terminal daemon before restarting.

    On point 5: I don't record per-process RSS, but node_exporter covers the host (an LXC container with 12.9 GB). Around the hang that started at 12:04:24 UTC, from 11:30 to 12:20:

    value
    MemAvailable flat at 10.7 GB (10.4 GB after the restart)
    Swap used flat at 0.21 GB
    Memory PSI (stalled) 0

    So the host was not under memory pressure when orcad stopped. The 4.4G figure is orca-serve.service's peak over its whole lifetime (5h20 of CPU), not necessarily its size at 12:04. That seems to point away from a host-level near-OOM. It doesn't rule out a V8 heap-limit stall inside orcad.

    I'll post the captures as soon as it hangs again.

  4. 1 remaining item

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

os:linuxIssues affecting Linux

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions