fix(client): a busy backend no longer looks like a disconnect

[btsouth]

Fixes #7231.

Problem

When the app comes to the foreground, the supervisor probes the live session with a 15s deadline. A probe that misses that deadline was treated exactly like a definite failure: the lease was replaced and the environment dropped to "Failed to connect. Reconnecting...".

The socket was still open and no close event had arrived. The backend was busy, not gone. The visible cost is the composer, which is disabled while the client recovers, and it lands precisely when someone has returned to the app to send a message.

Fix

On desktop and web, a missed deadline now waits 5s and probes once more. Only a second miss replaces the lease.

The retry lives inside the forked probe effect rather than in the signal loop, which matters for two reasons:

• Disconnect, explicit retry, going offline, and resume signals still interrupt it exactly as before, through the existing Fiber.interrupt(probe) paths.
• Recovery stays bounded. An earlier version of this fix tolerated a timeout and waited for the next application-active wakeup to decide. On desktop that wakeup only fires on visibilitychange, so a user who stays in the window would never produce one, and a backend wedged at the RPC layer (socket healthy, pings answered, handler deadlocked) would have sat at "connected" indefinitely with every request hanging. Doing the retry inline caps that at roughly 35s instead.

Switching the desktop path to Effect.timeoutOption also removes the guesswork about whether a given failure was our own deadline: a missed deadline arrives as None on the success channel, and every real probe failure still fails immediately. That matches the idiom already used by the provider probes on the server side.

Mobile is unchanged. application-active-probe keeps its 3s fail-fast and does not retry. After a background suspension the OS has usually killed the socket for real, so waiting longer would only delay recovery. Mobile cannot emit the bare application-active reason, so the retry path is unreachable from it.

Worth noting for reviewers: a genuinely dead transport is still detected independently of this path by the RPC pinger through session.closed, so nothing here is the sole defence against a dropped socket.

Relationship to #3553, #4137, and #5198

#3553 reported this and was closed by #4137, which made the probe itself lightweight (serverProbe rather than serverGetConfig). That was the right fix for probe cost. It did not change the teardown rule, and on a host that is starved rather than merely doing extra work a cheap probe misses its deadline too.

#5198 proposed tolerating a transient timeout and had the right instinct. It has carried merge conflicts since 1 Aug. This PR takes the same starting point and adds the retry, without which tolerance alone can strand a wedged backend.

Backend starvation itself is tracked in #4773. This is the client's reaction to it, which is worth fixing separately because the client cannot assume the server will ever be fast.

Testing

• Reworked the existing "stalled desktop foreground probe" test, which asserted teardown on the first timeout. Its 14999ms / 1ms deadline-boundary assertions are preserved on the retry.
• Added a test that the retry answering keeps the session, the generation, and the composer path untouched.
• 36/36 supervisor tests pass, over 5 consecutive runs.
• typecheck and vp lint packages/client-runtime/src/connection clean.

Docs: docs/internals/connection-runtime.md "Wakeups" section now states the retry, the bound, and the mobile exclusion.

Model: Claude Opus 5; harness: Claude Code.

---

Note

Medium Risk
Changes connection supervisor wake/probe policy on desktop/web, which affects when sessions are replaced and the composer is disabled; behavior is well-tested but touches core connectivity logic.

Overview
Desktop/web foreground health checks no longer tear down the live session on a single 15s probe timeout. A miss is treated as a busy backend (socket still open); the supervisor waits 5s, runs one more probe, and only then fails and replaces the lease. Real probe failures still reconnect immediately; mobile application-active-probe stays 3s fail-fast with no retry.

monitorConnectedLease switches the desktop path to Effect.timeoutOption and runs the retry inside the forked probe effect so disconnect/offline/resume signals still interrupt via existing Fiber.interrupt paths, with worst-case recovery bounded (~35s).

Docs and supervisor tests cover retry success (stay connected), double stall (reconnect), and preserved mobile behavior.

^{Reviewed by Cursor Bugbot for commit 1164990. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Fix desktop/web foreground probe to retry once before treating a stall as a disconnect

• A single missed 15s probe on desktop/web no longer immediately reconnects; the supervisor now waits 5s (CONNECTION_PROBE_RETRY_DELAY) and issues a second probe before treating the timeout as a failure.
• Mobile (application-active-probe) retains the original behavior: 3s deadline, no retry, immediate failure on timeout.
• Behavioral Change: desktop/web clients on a slow/busy backend will stay in the current session instead of re-establishing a new lease on the first stalled probe.

^{Macroscope summarized 1164990.}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Path: .coderabbit.config.ts
• Review profile: CHILL
• Plan: Advanced
• Run ID: 71b48ff1-b040-4f65-a2aa-53490d06589d

📥 Commits

Reviewing files that changed from the base of the PR and between d81afa0 and 098103f.

📒 Files selected for processing (3)

• docs/internals/connection-runtime.md
• packages/client-runtime/src/connection/supervisor.test.ts
• packages/client-runtime/src/connection/supervisor.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The connection supervisor now retries a timed-out desktop or web foreground probe once after five seconds before reconnecting. Mobile probes and explicit retries keep their existing deadlines. Tests cover retry outcomes and interruptions.

Changes

Foreground Probe Retry

Layer / File(s)	Summary
Foreground probe retry and validation packages/client-runtime/src/connection/supervisor.ts, packages/client-runtime/src/connection/supervisor.test.ts, docs/internals/connection-runtime.md	After an initial application-active probe timeout, the supervisor waits five seconds and starts one retry. A second timeout follows the existing reconnect path. Tests cover a successful retry, a stalled retry, explicit retry signals, and disconnects during the delay. Documentation describes desktop, web, and mobile probe behavior.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to 09810

This change lets a busy backend survive one missed foreground health probe without replacing the session. Mobile behavior is unchanged. I found no concrete merge-blocking risk.

Security Architecture Review

Security architecture risk: ⚪ Minimal · up to 09810

The additional wait is bounded and reuses the existing connection. Disconnects, session closure, and relevant account changes still end the connection, while definite probe failures remain immediate. No material security risk was identified in the changed behavior.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — A nonresponsive connected peer can prolong the client's connected-state display through the additional retry window. The changed effect is confined to the existing environment lease; the retry itself adds no target selection, cross-environment access, or privilege grant.

Trust Boundaries and Controls

• observed — Explicit disconnect, long mobile resume, and credential changes affecting an active relay lease are still checked during the retry window and interrupt the current probe. Session closure independently races the monitor, so the additional tolerance does not postpone a reported closure.

Resilience and Maintainability Implications

• observed — The first probe is interrupted before the delayed child is created, and terminal signal paths interrupt that same child reference. The attempt runs in a scope and clears exposed session and prepared-connection references on exit. Added tests assert retained session identity on successful retry and cancellation without another probe after disconnect; these assertions were inspected, not executed.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check		The title clearly summarizes the main change: a busy backend no longer causes the client to treat the live session as disconnected.
Description check		The description is complete and relevant. It explains the problem, the desktop/web retry behavior, the unchanged mobile behavior, scope, related issues, implementation constraints, documentation chang…
Linked Issues check		Issue #7231 requires the client to keep a live desktop/web session after one foreground probe timeout and to avoid unnecessary composer disruption. The reviewed changes add one retry after 5 seconds, …
Out of Scope Changes check		The reviewed changes are limited to the connection supervisor policy, its automated tests, and the connection-runtime documentation. These changes directly support issue #7231. No unrelated implementa…

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Approved at 098103f

Macroscope's review found this PR approvable — This is a focused connection-supervisor bug fix: desktop/web foreground probes retry once before reconnecting, while mobile and explicit-failure paths remain unchanged. The behavior is covered by targeted tests, and the change introduces no schema, infrastructure, security, billing, or static-analysis configuration impact.

Notes:

• Macroscope's correctness review did not run, so approvability was decided on eligibility alone.

^{You can add or adjust custom eligibility rules. Learn more.}

[macroscopeapp]

The new warning logs target.label verbatim, but connection labels are unrestricted strings. Please keep log annotations bounded and safe by recording the environment ID without the label.

Suggested change

	Effect.annotateLogs({
	"environment.id": target.environmentId,
	"environment.label": target.label,
	}),
	Effect.annotateLogs({
	"environment.id": target.environmentId,
	}),

^{Posted via Macroscope — Effect Service Conventions}

[macroscopeapp]

Correction to my inline suggestion on supervisor.ts: EnvironmentId is also an unbounded string and can come from a persisted or remote descriptor. Removing only environment.label is insufficient; please replace both raw annotations with a bounded category such as "environment.kind": target._tag.

^{Posted via Macroscope — Effect Service Conventions}

[btsouth]

Note

Posted by an AI agent on behalf of Tyler.

Rebased onto current main. The timeout warning now only annotates environment.kind (target._tag), so neither the label nor the environment id is logged.