test: RC4 five-host Relay stability and real-ACK validation #164

Goal and release boundary

Prepare a new v1.0.0-rc.3 campaign for #161 and #163 after an exact candidate is reviewed. This issue is a test plan, not permission to merge draft PR #162, publish RC3, replace KR production services, alter Cloudflare/OCI/Caddy policy, or start a live run. RC2's four-hour run ended near minute 172 without a terminal complete record and had public Relay failures; it is not a passed release soak.

Candidate preflight

• Identify the exact merged source commit, tag, wheel/sdist hashes, control build and installed version on each host. Review fix(relay): diagnose route failures and safely retry setup timeouts #162's stage diagnostics and setup retry independently; retry-only success does not close bug: public Relay intermittently loses authenticated routes during RC2 multi-host validation #161.
• Require green release gate, artifact provenance, fresh wheel/sdist/extras installs, pip check, standalone/MCP smoke and no protocol/identity/state migration surprise.
• Preserve existing device identities, pairings, login approvals, journals, replay state and unknown outcomes. Back up and verify KR state before any approved runtime switch; use schema-compatible rollback only.
• Resolve investigate(relay): bound idle receiver reconnect gaps and stale-room attaches #163's mechanism or explicitly document it as unproven. Capture privacy-filtered receiver room/reconnect/attach evidence without payloads or credentials.

Live matrix after separate rollout approval

• Exact RC3 on Mac mini, MacBook, quintet-desktop native Windows plus WSL receiver, KR and US; verify local discovery/MCP and direct/SSH routes separately from public Relay.
• Forced public Relay probe-only routes in both Mac directions, US→Mac, Mac→Windows/WSL, and server-originated paths. Record stage, failure category, timing, diagnostic UUID/CF-Ray where available, receiver room/close event and origin TCP metadata; do not infer consumption from 200/101, attach or queued.
• Randomize/alternate probe order and direction so US-first and warmed-receiver advantages cannot masquerade as PoP effects. Keep one request per bounded window and do not replay ambiguous sends.
• Fresh unique actual-message ACK tokens at T0, T2 and T4 in dedicated live Claude/Codex/Antigravity targets where supported, including Windows native targets. An incoming prompt or permission wait is receipt, not an ACK; exact user/live-TUI response is required.
• A single uninterrupted four-hour run: every-minute public and KR-local health/resource samples; every-five-minute direct and Relay routes; one controlled Relay restart at T2 with recovery target <=60 s. Record all failures, changes and UTC/KST boundaries rather than silently resetting the clock.
• A coordinator/process-lifetime guard must produce a terminal complete only after elapsed >=14,400 s and the T4 network checkpoint is submitted. Unexpected exit produces an explicit incomplete/aborted result; it never opens completion-gated follow-ups.

Pass / fail

Pass requires no unexplained Relay failures, no duplicate native effects, exact required ACKs, intact auth/replay state, recovery after the planned restart, and independently checked terminal evidence for the full four hours. If any route fails or evidence is incomplete, keep #161 and stable v1.0.0 blocked, file/deduplicate a finding, and plan another explicit candidate/run. A normal single CF/direct comparison or a successful retry is insufficient proof of long-term stability.

Optional direct-origin-vs-CF A/B or the previously prepared 15-minute receiver wrapper requires a separate bounded approval and current safety checks; the aborted RC2 run did not satisfy the wrapper's original completion gate.

Independent review additions (2026-09-23)

For an RC3 candidate to distinguish receiver-room absence from attach-after-stall, the preflight and live harness should include:

• Client diagnostic history per attempt, not just the final attempt: allowlisted stage, reason, elapsed milliseconds and WebSocket close code where available. Keep retry-success routes in a separate degraded count; they do not pass the clean stability gate.
• Receiver metadata-only lifecycle: room open/admission/WSS elapsed, room close reason and age, attach receipt, peer-TLS outcome/elapsed, reconnect gap. Separate expected idle expiry from genuine failures rather than hiding both under a one-per-minute reconnect warning.
• Relay metadata-only attach/room closure: receiver WebSocket age, last-pong age if observable, and room-end reason counters. Correlate clocks and diagnostic IDs without recording identities, payloads, credentials or raw exception text.
• A fixture that injects >10 s admission delay and a stalled receiver leg, then observes a subsequent client/retry. Assert first and second attempt diagnostics remain distinct and that the test's invoke deadline exceeds the candidate's worst-case setup+retry bound. The old 35 s harness deadline may collapse useful diagnostics into an outer timeout.
• T2 planned-restart recovery must be measured per affected route; the RC2 run did not recover the US→mini route inside 69 s. Make the soak runner independent of the interactive coordinator process and require a durable terminal complete/aborted record.
• If doing an optional receiver-leg direct-origin-vs-CF A/B, instrument the above first and get separate approval for host/IP /32, duration, network exception and rollback.

These are acceptance and design inputs, not authorization to deploy instrumentation or to publish RC3. The cause of #161 remains unproven.