Repository navigation
[bug] Renderer→host-service fetches wedge after hours ('Failed to fetch 127.0.0.1'); server healthy; terminals never open — and a dead host-service is never respawned #5503
Description
Summary
After ~2.5 hours of normal use, all renderer→host-service HTTP requests begin failing client-side. Opening a terminal in a workspace hangs forever, and workspace mutations surface Failed to delete <name>: Failed to fetch (127.0.0.1:48505). The host-service itself is healthy the entire time — the failure is in the app's own network stack, and only a full app restart recovers. While diagnosing, I also found the desktop app does not respawn (or even notice) a dead host-service.
Environment
- Superset Desktop 1.13.1 (stable), macOS 26.5 (Darwin 25.5.0), Apple Silicon (M-series, 16 cores / 128 GB)
- 23 workspaces registered in host.db, ~20 concurrent agent sessions, heavy git activity (same workload as [bug] Idle git status churn drives high host-service and Defender CPU usage #4937)
Timeline / evidence
App instance started 17:48. By ~20:10 the UI could no longer open a terminal in any workspace ("no matter how long I wait"), and a workspace delete failed with Failed to fetch (127.0.0.1:48505).
Measurements taken while the failure was live:
| Check | Result |
|---|---|
curl http://127.0.0.1:48505/ from a shell (×5) |
HTTP 404 in <1 ms — server event loop healthy |
| host-service open fds | 100 (limit 256) — no fd exhaustion |
| host-service CPU / state | ~30%, normal polling churn, not wedged |
| pty-daemon / terminal-host | idle; daemon.log shows zero client connection attempts during the failed terminal opens — the requests never left the app |
| Renderer's network-service utility process | 23 ESTABLISHED connections to 127.0.0.1:48505 — exactly one per registered workspace |
| System | load ~10, 89% memory free — not resource pressure |
A freshly restarted app instance holds ~11 connections to host-service; the wedged one held 23 (= workspace count). This looks like long-lived per-workspace subscription streams plus high request volume (git status refreshes, #4937) eventually leaving Chromium's per-host socket pool with nothing free / all in-flight slots stuck, at which point every subsequent fetch to the tRPC endpoint queues or fails instantly with Failed to fetch. I can't see inside the network service, so treat the mechanism as a hypothesis — but "server sub-millisecond healthy + UI cannot fetch + restart fixes it" is solid.
Bug 2: host-service death is never detected or recovered
During diagnosis I sent SIGTERM to the app-spawned host-service process:
- The desktop app did not respawn it (waited 15+ s), showed no error, and every host-backed UI action just failed silently.
superset start(CLI v0.2.23) brought up a daemon on a new port and rewrotemanifest.json, but the running app never re-read it and stayed pointed at the dead port.- Only quitting and relaunching the app recovered (it spawned its own host-service again).
Given #3896 explicitly made terminal sessions survive host-service restarts, the desktop app supervising/respawning its host-service child (or at least re-reading the manifest / surfacing a reconnect banner) seems like the intended missing piece. Right now a host-service crash produces exactly the symptom set in this report, indistinguishable from the connection-pool wedge, with no user-facing signal.
Secondary observations from the same window (host-service.log, ~2 h)
Sharing these since they add continuous background load that plausibly accelerates the wedge; happy to split into separate issues:
- 2,443 failed
ghPR head lookups (~20/min, no backoff):gh api --method GET repos/<owner>/<repo>/pulls -f state=all -f head=<owner>:main ...fails withCommand failed/ empty stdout from host-service, yet the identical command succeeds in a login shell — likely a PATH/auth difference in the app's spawn environment. Repeated ~20/min forever with no backoff on failure. Events were dropped by the FSEvents client. File system must be re-scanned.logged repeatedly under agent-driven file churn — each drop forces a full rescan, compounding [bug] Idle git status churn drives high host-service and Defender CPU usage #4937.[ensureMainWorkspace] failed ... Repository is in detached-HEAD state ... will retry via startup sweeploops forever for jj (Jujutsu) colocated repos, where a detached git HEAD is by design, not an error state (169 occurrences in 2 h). A per-repo "give up / needs attention" state would stop the churn.port-scan sync: registered 1[45] unattached daemon session(s) for scanningre-registers the same orphaned (pre-restart) sessions every cycle.
What I'd hope for
- A floor/cap or recycling for renderer→host-service connections, or moving per-workspace subscriptions onto a transport that can't starve the fetch pool (WebSocket multiplexing / HTTP2).
- Supervision of the host-service child: respawn on exit + renderer reconnect, or an explicit "host connection lost" UI state.
- Backoff on the failing
ghlookups and the detached-HEAD sweep.
Activity
fix(desktop): respawn host-service after unexpected exit
yasyf commentedon Jul 8, 2026
Update: recurred within ~15 min of a fresh app instance — and it's the renderer's sync consumer, not the socket pool
The wedge recurred on a brand-new app instance after only ~15 minutes, which let me bisect it much more precisely:
Writes succeed; the UI never hears about them. While the UI's "new terminal" hung forever:
superset terminals create(CLI v0.2.23) against the same host-service completed in 0.82 s, terminalactive.terminal_sessionsrow was created for every "hung" UI click (11 rows in the 20:30–20:40 window from retries — allactive, all invisible to the UI).Renderer reload alone fully recovers: Cmd+R made every phantom terminal appear instantly. No app restart, no host-service restart. That pins the failure to the renderer-side sync consumer (it dies silently and never reconnects), while host-service keeps emitting fine.
Nasty side-effect: each invisible-but-successful click leaves an orphaned, never-attached session with a live shell. host-service's
port-scan syncline grew from 15 to 36 "unattached daemon session(s)" in minutes, and the shells accumulate under pty-daemon.Possible trigger observed in the same window: repeated
Events were dropped by the FSEvents client. File system must be re-scanned.bursts (12 since the app start, heavy agent-driven file churn).So the concrete asks sharpen to:
snehendu098 commentedon Jul 26, 2026
@Kitenite
I sent SIGKILL to the host-service pid from the manifest, then polled at 3, 8, 13, 20, 30 and 40 seconds. The coordinator notices the crash within a millisecond and logs both exited with code null signal SIGKILL and crashed (signal SIGKILL), removes the manifest, and then nothing respawns it. At 40 seconds the manifest still had not come back. The renderer stays alive so the app looks healthy, but workspaces and terminals are dead until you restart it by hand from the tray menu
I also hit this without a deliberate kill: my local Docker stack went down, host-service died with it, and it never came back once Docker was running again. So this is reachable in normal use, not just synthetically
The pieces to fix it look like they are already in place. host-service-coordinator.ts has the child.on("exit") handler with a crash branch that currently just calls alertChildCrashed, and startWithPreferredPorts (used by reset()) already respawns while preserving the port. That handler also returns early when
status === "stopped", so a deliberate stop would not be picked up by a respawn added in the same placeI have not tested Bug 1 (fetches wedging after a couple of hours) since it needs long uptime. Happy to send a PR for Bug 2 on its own if that is useful. Two decisions I would rather not make unilaterally: immediate respawn versus a backoff with an attempt cap, and whether the modal error box should still fire when a respawn succeeds, given dialog.showErrorBox blocks the main process and that feels heavy for something that then heals itself
Kitenite commentedon Aug 26, 2026
Triage note: half of this is fixed, half is not. Leaving it open for the unfixed half.
Fixed — the dead host-service is now respawned. #5978 (
168687bc2) addedapps/desktop/src/main/lib/host-service-respawn.ts. The coordinator tracks per-org respawn state with an attempt budget and backoff delay, and cancels a pending respawn on quit or restart so a scheduled timer cannot resurrect a child during shutdown. That covers the second thing you found — your SIGKILL test where the coordinator noticed the crash in a millisecond, removed the manifest, and then nothing came back.There is also now a bounded connection-level retry on the query path (
hostServiceQueryRetry, 3 attempts with linear backoff), so a query in flight during a host-service restart self-heals instead of settling into a permanent "Failed to fetch" that only a manual "Try again" clears.Not fixed — the original wedge. The primary symptom in this report is that after ~2.5h all renderer to host-service requests fail client-side while the server is provably healthy (
curlreturning 404 in under a millisecond, no fd exhaustion, pty-daemon seeing zero connection attempts). That is a wedge in the app's own network stack, and I cannot find anything that addresses it. A 3-attempt retry does not help a permanently wedged stack — it just fails three times instead of once.So the remaining question is the one your evidence pointed at: the renderer's network-service utility process holding ESTABLISHED sockets that never complete. Worth noting this may share a root cause with #6610, where a host-service wedges into a hot loop after hours of uptime — both are "healthy for hours, then permanently stuck" with a similar time-to-onset.