Repository navigation
Description
Operating system
macOS (client and server both macOS; server is a headless Mac mini)
Orca version
1.4.163 (source references below are from main @ e08eba6, 1.4.165-rc.0)
Details
Short summary
A desktop client paired to a remote Orca runtime over Tailscale disconnects repeatedly. The server-side liveness reaper terminates a WebSocket after one missed 15s ping, which is inside the normal blackhole window of a Tailscale path change (direct↔DERP failover, NAT rebind, endpoint change, Wi-Fi roam). Because each subscription is its own socket, one path hiccup terminates every stream at once, and jitterless reconnect backoff makes them all redial in lockstep.
This is about why the sockets die in the first place, which I think is upstream of #9092 (that one is about failing to recover after wake). Cross-linking rather than duplicating.
1. Server reaps on a single missed ping
src/main/runtime/rpc/remote-runtime-server-heartbeat.ts:40-64
for (const socket of clients) {
if (resumedFromPause) { this.alive.add(socket) }
if (!this.alive.has(socket)) {
socket.terminate() // ← no second chance
reaped += 1
continue
}
this.alive.delete(socket)
socket.ping()
}Sweep interval is 15s (ws-transport.ts:25). Each sweep clears the alive flag and pings; the next sweep terminates if nothing inbound arrived. So the budget is 15s of silence, one strike, and the kill is terminate() — RST, no close frame, no close code, so the client cannot distinguish it from a network fault.
The resumedFromPause guard (elapsed > intervalMs * 1.5) correctly handles server sleep, but does nothing for ordinary transient packet loss.
Client-side monitors are more forgiving, so the server always wins the race:
- desktop: ping 10s, dead at 25s —
src/shared/remote-runtime-socket-liveness.ts:11-14 - browser: 25s idle → probe, 20s grace —
src/renderer/src/web/web-runtime-client.ts:68-70
The comment at remote-runtime-socket-liveness.ts:12-13 says the client timeout is "just under two server heartbeat periods (15s)" so both sides detect on a similar horizon. But the server only allows one period, so the two are not symmetric — the server kills at 15s while the client is still waiting until 25-30s.
There is no way to tune this in a deployment: heartbeatIntervalMs is documented test-only (ws-transport.ts:32) and there is no env override (unlike ORCA_HANG_WATCHDOG_TIMEOUT_MS etc).
2. Why Tailscale in particular
Tailscale path changes routinely blackhole 5-20s: direct↔DERP failover, NAT rebind, endpoint change, Wi-Fi roam, server wake re-key. That lands squarely on a 15s single-strike budget. WireGuard recovers, but the socket is already terminated. A relayed (DERP) path makes it much more frequent, and DERP is common for private servers behind symmetric NAT.
3. Per-subscription sockets amplify one hiccup into a storm
Each subscription opens a new WebSocket + new TCP + full X25519 handshake:
src/renderer/src/web/web-runtime-client.ts:132—new WebRuntimeClient(this.pairing)persubscribe()src/shared/remote-runtime-client.ts:569—new WebSocket(pairing.endpoint, ...)persubscribeRemoteRuntime
Only files.watch shares the main socket (SHARED_CONNECTION_SUBSCRIPTION_METHODS, web-runtime-client.ts:66). Terminals, browser screencast, and terminal multiplex each get their own socket and their own independent 15s liveness domain.
Consequences:
- one path failover terminates all of them simultaneously, so the user sees the whole workspace drop rather than one stream glitch
- they then all reconnect at once.
RECONNECT_DELAYS_MS = [500, 1000, 2000, 4000, 8000, 15_000](web-runtime-client.ts:65) has no jitter, so N sockets redial in lockstep with N simultaneous handshakes — which is itself more likely to lose a packet on a marginal link - with enough worktrees × terminals this approaches
MAX_WS_CONNECTIONS = 128(ws-transport.ts:11), where the server closes with 1013 andterminate()s after 1s (ws-transport.ts:209-215)
4. Delivery-credit stall resets the stream instead of retrying the frame
Observed on my client (main.trace.ndjson, redacted):
{"breadcrumb.name":"remote_terminal_stream_stall_recovery",
"reason":"delivery-credit-timeout",
"inactiveForMs":30004,
"outstandingDeliveryBytes":32,
"pendingAckBytes":0,
"resyncAttempts":0,
"snapshotPending":false,
"streamId":1}32 bytes of terminal output were stranded in the credit/ack flow control and never completed delivery. After 30s (remote-terminal-stream-watchdog.ts:2) the watchdog sets disposed = true and the multiplexer resyncs the whole stream (remote-runtime-terminal-multiplexer.ts:369). A single dropped binary frame across the tunnel is enough to strand the credit, and the recovery path is a full stream reset rather than a retransmit of the outstanding delivery.
Separately, that watchdog uses plain wall-clock setTimeout with no pause/sleep guard, unlike remote-runtime-server-heartbeat.ts:44 and remote-runtime-socket-liveness.ts:48 which both rebaseline on a suspiciously long tick gap. Closing the laptop lid mid-stream should therefore trip delivery-credit-timeout immediately on wake. (The instance above was genuine 30s network silence, not sleep — inactiveForMs is 30004, not hours.)
How to reproduce
- Run an Orca server on a Mac mini, reachable over a Tailscale address.
- Pair a desktop client from another machine on the same tailnet, ideally one where
tailscale statusshowsrelayrather thandirect. - Use remote terminals normally. Roam Wi-Fi, or force a Tailscale path change.
- Streams drop together and reconnect; repeat over hours.
Confirmation that it is server-side reaping: [ws-transport] heartbeat reaped N; M tracked sockets appears in the server log at the disconnect timestamps (remote-runtime-server-heartbeat.ts:66).
What I ruled out on my side
Getting the tailnet onto a direct path made disconnects much rarer, which is consistent with the diagnosis above but does not remove the cause:
- pinned
tailscaled --porton the server and forwarded that UDP port, sotailscale statusnow reportsdirectinstead ofrelay(24-75ms RTT) - server sleep disabled (
pmset) - verified
netcheckNAT/UDP health
A roam or a brief loss burst on a direct path still fits inside a 15s single-strike window.
Suggested fix direction
- Require 2-3 consecutive missed pings before
terminate(), or make the reap threshold a multiple of the ping interval, so a single lost control frame is not fatal. This alone would absorb most Tailscale path changes. - Keep the server threshold strictly looser than the client's detection window so the client observes a clean close instead of an RST, and make it configurable via an env override for high-latency links.
- Add jitter to
RECONNECT_DELAYS_MSso simultaneous drops do not redial in lockstep. - Multiplex more subscription types onto the shared socket (as
files.watchalready does) so a single transport fault does not fan out into N independent deaths and N handshakes. - Retransmit stranded delivery credit before resorting to a full stream resync, and give the delivery watchdog the same pause/sleep rebaseline guard the two liveness monitors already have.
Happy to test a patch against this setup — I can reproduce it fairly reliably.
Activity
daniel-rudaev commentedon Aug 4, 2026
Some controlled measurements from a second setup, in case they help narrow this.
Same shape as yours: Orca 1.4.163, headless Linux server (
orca serve --pairing-address <100.x> --mobile-pairing --port 6768), iOS client, both ends on the same tailnet.Four runs, phone only, nothing else touching the server, timed from the server journal:
reaped 1; 1 tracked sockets)Runs 1 and 2 separate two things that are easy to conflate: interaction is not what keeps the socket alive, since the idle-but-awake run survived just as long. Pongs alone held it for 3m40s.
Run 3 is the one that reproduces. Note the 41s: that is longer than two full 15s sweeps, so a two-strike reaper would still have terminated this socket. iOS suspends for as long as the screen is off, so extending the server's patience only shifts the kill later — it does not prevent it for a locked phone.
One thing I did not see mentioned above:
handleConnectionregisterspongandmessageas proof of life, but notping. Clients run their own liveness probe (REMOTE_RUNTIME_SOCKET_PING_INTERVAL_MS = 10_000), so a client that is provably alive and pinging is still counted as silent. That looks unintentional and is independent of the strike-count question.Also worth noting for anyone triaging from user reports:
withRemoteRuntimeTailscaleHintappends Tailscale guidance to any unreachable-family error whose endpoint is in 100.64.0.0/10. Every tailnet-paired setup therefore surfaces a Tailscale-flavoured message on any abrupt close, which makes reaper terminations read as network faults.Separately: on this setup the symptom is episodic. A cluster of 59-96s session lifetimes earlier the same day did not reproduce in any of the four runs above a few hours later, with no configuration change in between.
fix(runtime): require consecutive missed probes before reaping a pair…
fix(runtime): require consecutive missed probes before reaping a pair…