feat(gateway): persist expiring supervisor-presence leases for HA readiness

[drew]

Problem Statement

PR #2153 centralizes public sandbox readiness in the gateway, but supervisor sessions remain process-local. Its narrow HA safeguard preserves an existing shared Ready phase when a reconciling replica does not own the session. That prevents cross-replica demotion, but it cannot bound stale readiness if the owning gateway exits before writing an explicit disconnect.

A multi-gateway deployment needs cluster-wide, failure-aware evidence that a supervisor session is alive. Without it, SandboxPhase::Ready can remain stale after an owner crash or become dependent on which replica processes a backend snapshot.

Related work: #1951 and #2153.

Proposed Design

Persist an expiring supervisor-presence lease in the shared gateway store for each sandbox. The record should include at least:

• sandbox ID
• opaque supervisor session ID
• owning gateway replica ID
• expiry or last-renewed timestamp
• resource version or generation for compare-and-swap updates

When a gateway accepts a supervisor session, it should register the session locally for relay traffic and create or replace the shared lease. Heartbeats should renew the lease before expiry. Normal disconnect cleanup should remove the lease only when the stored session ID still matches, so cleanup from a superseded session cannot delete its replacement.

Gateway readiness composition should consult a valid cluster-wide lease, directly or through a watch-backed local cache, instead of treating absence from the process-local SupervisorSessionRegistry as proof of disconnection. Terminal backend states (Error and Deleting) continue to take precedence.

If an owning replica crashes, the lease expires without an explicit disconnect. Reconciliation then moves the sandbox to Provisioning with Ready=False, reason SupervisorNotConnected. A reconnect on another replica replaces or renews presence and promotes the sandbox normally.

The design should define clock/TTL behavior, renewal cadence, takeover semantics, and cleanup of expired records. It should reuse existing store CAS and HA lease patterns where practical without coupling supervisor presence to reconciler ownership.

Alternatives Considered

• Preserve an existing shared Ready phase when the local replica lacks the session. This is the narrow safeguard in refactor(compute): make sandbox readiness gateway-owned across all drivers #2153 and prevents incorrect cross-replica demotion, but readiness can remain stale after an owner crash.
• Route all supervisor and reconciliation traffic for a sandbox to the same replica. This introduces affinity and failover constraints and still needs a failure detector.
• Write presence without expiry. This propagates cluster-wide ownership but cannot recover safely from abrupt process failure.

Agent Investigation

The readiness snapshot path in #2153 checks the process-local supervisor registry and writes the composed phase to shared sandbox state. Issue #1951 explicitly identifies persisted or leased supervisor presence as the more complete HA solution.

Existing-issue searches for supervisor session lease readiness, HA supervisor presence, and stale Ready gateway replica found #1951 plus related session/lifecycle work in #1731 and #1732 and the completed reconciler-lease work in #1429. None provides an expiring, per-sandbox supervisor-presence record.

Definition of Done

• A live supervisor session has a cluster-visible, expiring presence record.
• Heartbeats renew presence without unbounded writes or expiry gaps under normal operation.
• Superseded-session cleanup cannot delete the current session's record.
• Readiness composition uses valid cluster-wide presence rather than a negative process-local lookup.
• Owner crash or network loss causes readiness to demote within a documented bound.
• Reconnect on another replica safely replaces expired or superseded ownership.
• Unit tests cover renewal, expiry, supersession, and stale cleanup.
• HA E2E coverage places the supervisor session and reconciler on different replicas and exercises owner failure.
• Architecture documentation describes the lease lifecycle and readiness guarantees.