Host service stops answering its health check under session load; SQLite WAL never checkpoints #7426

Activity

1. bade-paygees

   added

   bugSomething isn't workingSomething isn't working

   on Sep 11, 2026
2. github-actions commented on Sep 11, 2026

   

   github-actionsBot

   on Sep 11, 2026 – with GitHub Actions

   Contributor

   More actions

   Summary

   Two claims here, and they need to be separated — the code says one of them is a real (if minor) defect and the other is a misreading of normal SQLite behaviour, which means the actual cause of the wedge is still open and almost certainly lives in the event loop rather than in the database.

   The health endpoint touches no SQLite at all, so WAL lock contention cannot be what's starving it. But the "WAL survives a clean superset stop byte-identical" observation is a genuine bug in the standalone entry point, just a different one than reported.

   ---

   1. The WAL is not the cause — the numbers say it checkpoints fine

   packages/host-service/src/db/db.ts:35-37 opens the DB with journal_mode = WAL and busy_timeout = 5000, and never sets wal_autocheckpoint or journal_size_limit. That leaves SQLite's default wal_autocheckpoint = 1000 pages in force.

   Run the reported file sizes against that default:

   • host.db = 450,560 = 110 × 4096 → page size is the 4096 default.
   • WAL frame = 24-byte header + page = 4120 bytes; WAL file = 32-byte header + n frames.
   • (4,132,392 − 32) / 4120 = exactly 1003 frames.

   A WAL sitting at 1003 frames with a 1000-page autocheckpoint threshold is the signature of checkpointing working: SQLite checkpoints at the threshold, then rewinds and reuses the same file rather than truncating it, so the file parks at its high-water mark and stays there. An uncheckpointed WAL would not stop at 1003 — it would be at whatever the workload wrote, and it would keep climbing. The reported size is stable at the threshold, which is evidence against unbounded growth, not for it.

   The futex_wait_queue / 11.5% CPU sampling is also not consistent with SQLite lock contention: better-sqlite3 is synchronous and takes locks on the main thread, and busy_timeout = 5000 would surface as SQLITE_BUSY throws in the log long before it looked like this.

   2. The real bug behind "byte-identical across a clean stop/start"

   That part of the report is correct and it's a genuine defect, confined to the standalone/CLI entry point:

   • packages/host-service/src/app.ts:371-418 — createApp() returns a dispose() that closes the SQLite handle ($client?.close() at app.ts:413). Closing is what performs the final checkpoint and lets the WAL be truncated.
   • packages/host-service/src/serve.ts:79 — the standalone entry destructures { app, injectWebSocket, api, db, launchSandboxAgent } and never takes dispose.
   • packages/host-service/src/serve.ts:102-124 — the only signal handlers are inside if (isDev). In production there is no SIGTERM/SIGINT handler at all.
   • packages/cli/src/commands/stop/command.ts:26 — superset stop sends SIGTERM, waits 10s, then SIGKILL.

   So on a self-hosted box, superset stop delivers SIGTERM, Node's default action kills the process immediately, the SQLite handle is never closed, no checkpoint-and-truncate runs, and the WAL file is found at exactly the same high-water mark on restart. Precisely what was observed.

   The desktop path does not have this bug — apps/desktop/src/main/host-service/shutdown.ts:69-100 runs dispose() under a deadline before exiting. Only the CLI/systemd path skips it, which matches the reporter running the host standalone on Ubuntu.

   This is worth fixing on its own merits (clean shutdown, no recovery work on next open), but it is not the wedge.

   3. Where the health-check starvation most likely originates

   health.check is deliberately cheap and DB-free:

   • packages/host-service/src/trpc/router/health/health.ts:9-24 — returns pid/version/registration state, reads no tables.
   • packages/host-service/src/app.ts:348-367 — createContext calls hostAuth.validate(), which for a self-hosted PSK host is a local timingSafeEqual (packages/host-service/src/providers/host-auth/PskHostAuthProvider/PskHostAuthProvider.ts:11-25). No network, no DB.
   • publicProcedure carries only a Sentry middleware (packages/host-service/src/trpc/index.ts:102-106).

   With a 2s client timeout (packages/cli/src/lib/host/health.ts:18), the only way this fails while the process burns CPU is event-loop starvation. The second "what would help" item is therefore right in spirit but wrong in mechanism: the responder is already independent of session bookkeeping; it's the shared loop that's contended.

   The strongest candidate, and the one that matches the reporter's own correlation with terminal count and the exact log line they quoted, is the port scanner:

   • packages/host-service/src/terminal/reaper/reaper.ts:242-267 — this is the source of port-scan sync: registered N unattached daemon session(s). Every reap pass registers each unattached daemon session into the port manager, so registered sessions grow with terminal count.
   • packages/port-scanner/src/port-manager.ts:18 — SCAN_INTERVAL_MS = 2500. Every 2.5s, for every due session.
   • packages/port-scanner/src/port-manager.ts:311-355 → packages/port-scanner/src/procfs.ts:177-230 — buildInodeToPid fans out fs.readdir('/proc/<pid>/fd') across every pid in every session tree with no concurrency cap (procfs.ts:185-186; only the inner readlink is limited, to 64 at procfs.ts:183). With 148 terminals running agents, that's a session-tree pid set in the hundreds-to-thousands, each contributing a readdir plus one readlink per fd, onto libuv's 4-thread pool.
   • packages/port-scanner/src/detached.ts:57-63 — collectDetachedPids then reads /proc/<pid>/environ for every process on the machine that is not in a session tree (procfs.ts:302-329, 64-wide). On a 12-core box hosting 60 workspaces that is the whole process table, every scan.
   • packages/port-scanner/src/procfs.ts:269-277 — readProcessName is awaited serially inside the listener loop, one /proc/<pid>/comm read at a time per distinct listening pid.
   • packages/port-scanner/src/port-manager.ts:501-506 — a scan that had a follow-up queued tail-recurses into another scan immediately. Under continuous agent output across 148 terminals, hint-triggered scans (port-manager.ts:206-212) can chain back-to-back with no idle gap, which would read as steady low-double-digit CPU indefinitely.

   Secondary contributors on the same loop: packages/host-service/src/terminal/reaper/reaper.ts:163-175 does a synchronous full-table terminal_sessions read (better-sqlite3 blocks the loop) on every pass, and the per-workspace git/PR sweeps at packages/host-service/src/runtime/pull-requests/pull-requests.ts:333-338 fan out across all 60 workspaces every 5 minutes. packages/host-service/src/no-main-loop-blocking.test.ts shows this failure mode is already a known concern in this package — but that ratchet only polices sync subprocesses and on-loop git, not the procfs fan-out above.

   One smaller thing worth noting while someone is in this file: createLimiter at packages/port-scanner/src/procfs.ts:149-170 re-checks active >= concurrency only on entry, and a released waiter increments active a microtask later — so a burst can admit more than concurrency tasks. It over-admits rather than deadlocks, but it means the "bounded" 64 is soft exactly when the machine is busiest.

   Reproduction

   For the shutdown/WAL defect (deterministic, cheap):

   1. Start a standalone host: superset start --daemon.
   2. ls -l ~/.superset/host/<org>/host.db-wal.
   3. superset stop, then re-check the WAL size — it will be unchanged, because SIGTERM killed the process without closing the handle.
   4. With a fix, a clean stop should leave the WAL at 0 bytes (or absent).

   Note that a bun:test won't work for this path — packages/host-service/src/app.ts:70-76 records that better-sqlite3 isn't loadable under Bun and tests substitute bun:sqlite. A .node-test.ts (the convention already used in this package) or the shell check above is the right vehicle.

   For the wedge: the cheapest signal is loop lag, not another ps sample. On the wedged host, either:

   • run the service with --cpu-prof / attach node --inspect and take a 10s CPU profile while health.check is timing out, or
   • add a temporary loop-lag probe (setInterval measuring drift against SCAN_INTERVAL_MS) and log it alongside the scan.

   If the theory above is right, the profile will be dominated by buildInodeToPid / readEnvValuesLinuxProcfs / their callbacks, and lag will track registered-session count. A cheap confirmation the reporter can run today: set UV_THREADPOOL_SIZE=32 (or higher) on the host-service process and see whether the interval between wedges lengthens. That isn't a fix, but it's a strong discriminator — if it helps, the cause is procfs fan-out, not SQLite.

   Suggested fix direction

   1. Shutdown (small, self-contained). Take dispose from createApp() in serve.ts:79 and install production SIGTERM/SIGINT handlers that run it under a deadline, mirroring createShutdown in apps/desktop/src/main/host-service/shutdown.ts. That single change makes superset stop close SQLite and checkpoint. Optionally add sqlite.pragma("journal_size_limit = <N>") in db.ts so the WAL is truncated at each checkpoint rather than parking at its high-water mark — cosmetic, but it removes this whole diagnostic red herring for the next person.

   2. Port scanner (the load that actually scales). Bound the per-scan work independently of session count: cap concurrent /proc/<pid>/fd readdirs the way readlinks already are; make the scan cadence or the per-tick session budget degrade as registered sessions grow, instead of fixed 2.5s over everything due; and consider whether collectDetachedPids needs to sweep the entire process table on every tick rather than on a slower cadence. Longer term this is a natural candidate for the existing worker pool (packages/host-service/src/workers/) so it stops competing with tRPC responses at all.

   3. Health responsiveness. Even with the above, it's worth making loop lag observable — log a warning when a scan tick drifts past some multiple of its interval, so the next occurrence self-diagnoses instead of requiring ps/WCHAN forensics.

   To the direct question: I read current main for this triage and both the missing-dispose shutdown path and the unbounded procfs fan-out are present there, so I can't tell you that 1.27 or 1.28 fixes either. Someone with release history should confirm against the actual 1.27/1.28 tags before you upgrade expecting relief. In the meantime the superset stop && superset start --daemon workaround is sound for exactly the reason described — pty-daemon is a separate PPID-1 process tree by design (serve.ts:98-101).

   Thank you for the unusually precise report — the pid sampling, the fd counts, and especially the interval-versus-session-count progression are what make the event-loop explanation testable.
3. drewtang commented on Sep 11, 2026

   

   drewtang

   on Sep 11, 2026

   More actions

   Reproduced again with both client and host on 1.28.0. The process remained alive, but its local health endpoint timed out and the listener backlog reached 390/511. The VM had 69 GiB available memory and no OOM or kernel errors.

   This confirms that the load reductions in #6848 and #7213, and the relay recovery in #6724, do not fully address the failure. Restarting only the host service restores access temporarily without interrupting the running PTY sessions.