[Bug]: Every server restart takes T3 Connect remote access down for 1–1.5 minutes: HTTP and the tunnel both wait for DB recovery, and the client backs off up to 5 minutes #24661

Summary

On a self-hosted t3 serve linked to T3 Connect, a restart (update, settings change, systemctl restart) cuts remote clients off for about 1–1.5 minutes plus the client's backoff. The process itself is down for only ~9 s. The rest is:

1. every HTTP route waits on startup readiness, which comes only after V2 orchestration recovery, and recovery grows with the state DB;
2. the managed cloudflared is a child of the server and is started only after that same recovery, so the tunnel is gone for the whole window (the relay answers 502);
3. client backoff keeps growing during the window (cap 300 s), so after a ~90 s outage the next try is often 64–128 s away.

Version / setup

• 0.1.3-nightly.20261023.3566, Linux x86_64, systemd user service via t3 __service-launcher. Same code on main at edb2b2f365.
• T3 Connect managed tunnel (cloudflared 2025.x from PATH). Clients: macOS desktop nightly over the relay.
• statev2.sqlite: 25–26 GB on host A, 16 GB on host B.

Numbers (server log, host A, n=4 restarts)

• Stop → listen: ~9 s (10-24: stop 13:58:12, listen 13:58:21).
• cloudflared registers 4/4 connections < 3 s after spawn, so almost all of the gap is server readiness, not Cloudflare.
• Host B (16 GB DB): listen → relay 40 s (n=1).
• What users see: ~1.6–3.7 min per restart including backoff.
• Host A restarted 4 times in 7 days (updates, config-change restarts, manual).

Where in the code

• Global readiness middleware: every request waits on awaitCommandReady (server.ts:685-690). Readiness is signalled at serverRuntimeStartup.ts:670, after "V2 orchestration recovery completed" (:574).
• Tunnel start: server.ts:921 (startManagedTunnelIfOriginConfirmed), which in practice runs 0.1–1.4 s after recovery.
• The connector is always a child: ManagedEndpointRuntime.ts:632-657 (detached: false, TUNNEL_TOKEN from stored config). Stopped on shutdown by the finalizer (:796). The only env knobs are TUNNEL_TRANSPORT_PROTOCOL (:212) and T3CODE_CLOUDFLARED_PATH (relayClient.ts:32).
• Client: RETRY_MAX_DELAY_MS = 300_000, 15 s establishment timeout (supervisor.ts:41-45).

Why an external connector alone does not fix it

A second cloudflared replica with the same token in its own service would save at most ~1.4 s. While the process is down the edge answers 502. During recovery the TCP connection is accepted but the request is held by the readiness middleware until the client's 15 s establishment timeout. Either way the client is in its retry loop with growing backoff. The fix has to be on the readiness path and the retry policy.

Asks (1 + 3 together fix most of it)

1. Answer health and connect before recovery finishes. Accept the WebSocket and RPC ping at listen time and report a "starting / recovering" state, instead of holding every route on awaitCommandReady. The client stays connected and does not back off.
2. Start the tunnel at listen time, not after recovery. Optionally support an external connector (a setting that skips spawning cloudflared, plus a way for a service manager to get the connector config).
3. Short backoff on planned restarts. Before a planned shutdown (update, settings restart), tell clients "restarting, retry in N s", and reset or cap backoff while the relay answers 502/530 for a known-restarting environment.
4. Optional: drain on SIGTERM with a restart close code, so the client can tell a planned restart from a network loss.

Related

• [Bug]: Web and desktop offer no retry while an environment is reconnecting, and the wait grows to five minutes #22024 (no retry while the wait grows to 5 min) and its PRs fix(web): offer retry while an environment reconnects #23194 / fix(web): offer retry while an environment is reconnecting #22207 (retry button). Those let a user retry by hand; this asks the server to be reachable sooner.
• fix(client-runtime): returning to the app no longer restarts the reconnect backoff for an unreachable host #24064 (backoff restarts on app focus).
• [Bug]: Requests that arrive during server startup are never answered (socket accepts, handler not attached yet) #12215 (closed): requests during startup were never answered. The readiness gate fixed that by holding them, which is the wait described here.

Workarounds today

Fewer restarts (quiet windows), TUNNEL_TRANSPORT_PROTOCOL=http2, shrinking the state DB.