Remote Linux host stays alive but becomes offline after relay WebSocket stalls #6229

Activity

1. github-actions commented on Aug 6, 2026

   

   github-actionsBot

   on Aug 6, 2026 – with GitHub Actions

   Contributor

   More actions

   Summary

   The report describes a host-service that logs no inbound traffic for ~77s, forcing reconnect and then never comes back: the process stays alive, superset-host.service stays active (running), and the control plane keeps the host offline until a manual systemctl --user restart. Reading the v1 tunnel client, this is consistent with a real defect: the inbound-silence watchdog's only recovery mechanism is a graceful WebSocket.close(), and every reconnect in that client is scheduled exclusively from the onclose handler. If the TCP path is blackholed (packets dropped rather than reset — which is what a stuck ESTAB with a 39 KB unacked send queue looks like), the close frame can never be flushed, the closing handshake never completes, onclose never fires, and nothing ever calls scheduleReconnect(). The client parks in CLOSING forever.

   Affected code

   • packages/host-service/src/tunnel/tunnel-client.ts:375-391 — the watchdog
   • packages/host-service/src/tunnel/tunnel-client.ts:123-156 — onclose, the only path that calls scheduleReconnect() for an established socket
   • packages/host-service/src/tunnel/tunnel-client.ts:400-418 — scheduleReconnect()
   • packages/host-service/src/serve.ts:108-117 — connectRelay() is called exactly once; there is no outer supervisor that re-drives the tunnel
   • packages/host-service/src/tunnel/connect.ts:22-36 — protocol negotiation; the [host-service:tunnel] prefix (not tunnel-v2) in the reported logs means this host is on the v1 client

   Suspected root cause

   Two lines do the damage together.

   // tunnel-client.ts:377-390
   this.watchdogTimer = setInterval(() => {
     if (!this.socket || this.socket.readyState !== WebSocket.OPEN) return;   // ← 378
     const silentFor = Date.now() - this.lastInboundAt;
     if (silentFor > INBOUND_SILENCE_TIMEOUT_MS) {
       console.warn(`...no inbound traffic for ${silentFor}ms, forcing reconnect`);
       try { this.socket.close(4002, "Inbound silence timeout"); } catch {}   // ← 385
     }
   }, WATCHDOG_INTERVAL_MS);

   1. Line 385 requests a graceful close. On a blackholed path the close frame sits in the kernel send queue (matching the reported 39 KB backlog) and the peer's close never arrives, so readyState goes to CLOSING (2) and stays there — potentially for the full TCP retransmission budget (tcp_retries2, ~15 min) or indefinitely. this.socket is never nulled and no reconnect is scheduled.
   2. Line 378 then makes the watchdog a permanent no-op, because readyState is now CLOSING, not OPEN. That explains why the logs show exactly one no inbound traffic line per occurrence with no follow-up — there is no escalation and no second attempt.

   Supporting evidence from elsewhere in the repo:

   • The relay is sending pings on schedule (answers your question 1): apps/relay/src/tunnel.ts:135-142, every 30 s, with a 3-missed-ping (90 s) server-side timeout. 75 s of host-side tolerance leaves room for two lost pings, so the timeout constant itself looks sane — the bug is what happens after it fires.
   • The maintainers already hit the "close frame doesn't reach the peer" failure mode from the other direction, documented at apps/relay/src/tunnel.ts:210-217: "the WS close frame doesn't reliably reach the host within Fly's kill_timeout window (host's TCP socket sees ESTABLISHED with no onclose for 75+ seconds)". The drain path was given an in-band {type:"drain"} message precisely because closing the socket is unreliable — but the watchdog path never got the same treatment.
   • The v2 client does not have this shape: packages/host-service/src/tunnel/tunnel-client-v2.ts:116-125 calls control.reconnect() on partysocket, which (as I read its contract) detaches and redials rather than waiting for a close event. So this is a v1-only regression surface.

   Answering your other questions from the code:

   • (2) Why 4002 doesn't produce a replacement connection: nothing schedules one. scheduleReconnect() is reachable only from onclose (:154), the connect-timeout deadline (:78), and the connect() catch/no-token paths (:94, :168) — none of which run here.
   • (3) Does the old socket block connect()? Not by readyState — the guard at :58-63 only bails on CONNECTING/OPEN, so a CLOSING socket would not block a manual connect(). And onclose is already guarded by if (this.socket !== socket) return; (:124), which means orphaning the wedged socket and dialing a new one is already safe today. The old socket is simply never abandoned.
   • (4) Verifying against control-plane registration: worth doing, and there is a known adjacent race documented in plans/20260804-1815-relay2-durable-objects-canonical.md:51-52 — a dying host's setOnline(false) landing after a replacement's setOnline(true). On the relay, unregister() is guarded against this within one instance (apps/relay/src/tunnel.ts:174), but not across Fly machines: if your host eventually reconnects to a different relay instance and the old instance's socket finally closes, apps/relay/src/tunnel.ts:184 writes setOnline(false) and clobbers the live tunnel's online state. That is a distinct failure mode with the same user-visible symptom, so it's worth ruling in/out (see diagnostics below).
   • (5) Exit after repeated failures: nothing in serve.ts does this today; your Restart=always unit would recover fine if it did.

   A contributing factor for the growing send queue: send() at :189-193 only checks readyState === OPEN and never inspects bufferedAmount, so PTY output keeps being written into a dead socket unbounded.

   Reproduction

   Deterministic (blackhole the path — use DROP, not REJECT; an ICMP reject would close the socket cleanly and not reproduce this):

   1. Start the host service and confirm connected to relay for host ….
   2. Note the peer address: ss -tnp | grep superset-host.
   3. sudo ip6tables -I OUTPUT -d <relay-v6> -p tcp --dport 443 -j DROP (and the matching INPUT rule).
   4. Wait ~80 s. Expected-if-buggy: exactly one no inbound traffic for …ms, forcing reconnect, no reconnecting in Xms (attempt N) line, and ss showing the connection still ESTAB with a growing Send-Q.
   5. Remove the rules. The host should not recover promptly — confirming the wedge.

   Unit-level bun:test sketch against TunnelClient with fake timers: inject a stub WebSocket whose close() sets readyState = 2 (CLOSING) and never dispatches onclose. Open the tunnel, advance the clock past INBOUND_SILENCE_TIMEOUT_MS + WATCHDOG_INTERVAL_MS with no inbound messages, then advance several more minutes and assert the stub's constructor was called a second time. Today it will be called exactly once.

   Diagnostics that would confirm which of the two failure modes you hit: during an outage, run ss -tn '( dport = :443 )' | grep <relay-ip>. One wedged connection with a large Send-Q and no reconnecting in … log ⇒ the watchdog wedge above. A fresh ESTAB connection with an empty queue plus a connected to relay line in the log, while the control plane still says offline ⇒ the cross-instance setOnline(false) clobber instead.

   Suggested fix direction

   In startWatchdog(), stop treating close() as recovery and stop gating the check on OPEN:

   1. On silence timeout, request the close and immediately abandon the socket: clear its handlers, set this.socket = null, stopWatchdog(), cleanupChannels(), then scheduleReconnect(). The this.socket !== socket guard at :124 already makes a late onclose from the orphan harmless. If the runtime exposes terminate(), prefer it over close() so the FD is actually destroyed rather than leaked until TCP gives up.
   2. Alternatively/additionally, arm a short "close did not complete" deadline (~5 s) after the 4002 close, mirroring the existing CONNECT_TIMEOUT_MS pattern at :66-79, and take path 1 when it expires. Also let the watchdog act on CLOSING, not just OPEN, so it can escalate instead of going silent.
   3. Add a backpressure guard in send() (:189): if bufferedAmount stays above a threshold for more than a few seconds, treat the path as dead and force the same reconnect. This would have caught the 39 KB stall long before the 75 s inbound timer.
   4. For the supervisor safety net you asked about: track consecutive failed reconnects (or time since the last open) and process.exit(1) past a bound — cheap, and it makes the existing Restart=always unit sufficient. Worth pairing with a re-registration check so an established-but-unregistered tunnel also counts as a failure.

   Interim workaround on your side: your log-triggered watchdog is the right shape; keying it on the no inbound traffic line without a following connected to relay within ~30 s will avoid restarting on benign transient reconnects.
2. zeroknots commented on Aug 6, 2026

   

   zeroknots

   on Aug 6, 2026

   Author

   More actions

   I checked out superset-sh/superset and found a concrete reconnect defect in the v1 host tunnel client that matches the observed failure.

   Root cause

   packages/host-service/src/tunnel/tunnel-client.ts:375-390 detects 75 seconds without inbound traffic and calls:

   this.socket.close(4002, "Inbound silence timeout");

   But every reconnect after an established connection is scheduled only by socket.onclose (:123-156). On a blackholed TCP path, the WebSocket close frame remains buffered, the socket stays in CLOSING, and onclose never fires. Therefore scheduleReconnect() is never called and the process remains alive but permanently unreachable.

   This matches the captured state: TCP remained ESTABLISHED with a ~39 KB send queue, the host logged forcing reconnect, no later reconnect log appeared, and restarting the process recovered immediately.

   The existing identity guard in onclose already makes immediate replacement safe:

   if (this.socket !== socket) return;

   So the watchdog should atomically orphan the stale socket before attempting the graceful close:

   1. Capture the stale socket.
   2. Set this.socket = null and this.connecting = false.
   3. Stop the watchdog and clean up local channels.
   4. Best-effort socket.close(...).
   5. Call scheduleReconnect() immediately rather than waiting for onclose.

   I implemented this locally with a regression test whose fake WebSocket enters CLOSING but deliberately never emits close. Before the change, no replacement can be scheduled; with the change, a replacement WebSocket is created. Focused test and Biome check pass.

   There may be a second, distinct cross-relay online-state race, but it is not needed to explain this reproduced stuck-CLOSING failure.
3. andyst-dev

   added 2 commits that reference this issue on Aug 13, 2026

   fix(host-service): force tunnel reconnect on inbound silence without …

   ...

   cc046ac

   fix(host-service): force tunnel reconnect on inbound silence without …

   ...

   efbf04f