Repository navigation
bug: public Relay intermittently loses authenticated routes during RC2 multi-host validation #161
Description
Summary
RC2 five-host revalidation (#150) reproduces intermittent no_authenticated_route on public Relay routes while forced-direct routes and KR origin health remain healthy. Treat this as an unresolved release gate, not as proof of credential expiry or a Windows regression.
Environment / exact artifacts
- Published
v1.0.0-rc.2, Python1.0.0rc2, commitfe9d8d83a8f9eb7bb26d47c6c3ebbf6c9b211608. - macOS ARM64 standalone + relay venv, Ubuntu ARM64 sender, native Windows core with Ubuntu WSL receiver.
- KR root-owned control built from the exact tag; Relay installed from published RC2 wheel. Same production identity, auth state and replay journal retained; no schema reset/migration.
- All RC1 campaign processes were stopped; only RC2 receivers and a new bounded campaign are active.
Reproduction
- Use an already enrolled, paired receiver started with
device serve --login --relay wss://relay.abruption.dev/v1/connect. - From a paired sender, run
list --device <receiver> --device-state <existing-state> --device-route relay --relay-login --json. - Compare with
--device-route directwhere the receiver's direct address is available.
Expected / actual
Expected: authenticated public Relay listing succeeds and idle receiver reconnects reliably.
Actual during 2026-09-22 RC2 preflight and T0: direct listing succeeds for US→Mac mini and both Mac directions. The same public Relay routes intermittently return ok=false, reason=no_authenticated_route, retryAllowed=false, consumptionConfirmed=false. Mini→WSL/native-Windows also fails public Relay preflight. The previous RC1 run first recorded similar route failures after minute 84, so this is not established as an RC2-only regression.
Redacted evidence / uncertainty
- KR control and Relay stay active with zero automatic restarts; health and resource samples are recorded throughout the new four-hour campaign.
- Client login metadata has over 21 hours remaining; login expiry is not supported by this evidence.
- A diagnostic using the actual CLI User-Agent successfully issued control challenge/admission and obtained Relay HTTP 200. Other attempts time out during control calls. Origin counters show some accepted sockets but no stable set of receiver waiting rooms.
- An early diagnostic that incorrectly used Python's default User-Agent received Cloudflare 1010/403. That diagnostic mismatch must NOT be used to claim that production session-peer is blocked by that rule.
- Native Windows Claude and Codex RC2 preflight exact ACKs are explicitly user-confirmed. This does not validate the failed public Relay path.
- No blind resend, state clearing, credential copying, edge bypass, security-rule change, or automatic rollback occurred.
Acceptance
- Isolate origin admission/WSS, edge transport, and client listener/reconnect causes with sanitized stage-level evidence.
- Explain and resolve the intermittent failures, then repeat the affected real-ACK routes and a clean four-hour run before stable promotion.
- Improve sanitized diagnostics if
no_authenticated_routehides the actionable stage.
RC2 has already been published; any required runtime fix needs a subsequent explicitly approved release candidate. This issue blocks v1.0.0 acceptance and does not authorize publishing or relaxing production security settings.
Activity
abruption commentedon Sep 22, 2026
RC2 T0 follow-up: public Relay US→Mac mini Claude succeeded with a separately observed exact live-TUI ACK, while the other five public-Relay T0 send attempts failed no_authenticated_route. Native Windows T0 submits succeeded but their new actual ACKs are still awaiting user confirmation. Minute-3 public health curl timed out (exit 28) while simultaneous KR origin health/resources remained healthy. This confirms intermittent behavior rather than a universal admission denial. The bounded run continues to preserve failures; no retry or edge/security change was performed.
abruption commentedon Sep 22, 2026
Isolated-worktree candidate is now available as Draft PR #162, commit
1ad07e4b9eb25e7f39bb8cfb7d6b696c6122cf0c(base: published RC2fe9d8d8). This does not close #161.Implemented sanitized per-stage connection diagnostics, rate-limited receiver reconnect warnings, and one timeout-only Relay setup retry before application submission. Auth/HTTP/TLS refusals are not retried; post-submit unknown outcomes are never replayed. A local real Relay/WebSocket/native-socket test verified exactly one native effect after the injected setup timeout.
Read-only production diagnostics reproduced public requests stalling after TCP/TLS success while origin health remained available. The sampled Cloudflare BIC event at 16:07 KST was the earlier diagnostic using Python-urllib/3.14, not proof of a production CLI block. A candidate forced-Relay list both succeeded and failed on separate attempts; the failure identified
relay_attach / connection_failed / attempts:2. The underlying public-path cause remains unresolved.Local verification: full suite 494 tests (11 skips, no failures), separately enabled built control integration 9 passed, fresh wheel/sdist content contract 3 passed, generator and diff checks passed. PR CI is running. The candidate is intentionally unpublished; no installed RC2 package, production Relay/control, Cloudflare/security setting, or campaign process was changed. Existing RC2 soak and reporter were confirmed still running.
Acceptance remains: resolve the remaining cause, obtain explicit approval for the next candidate rollout, repeat affected real ACK routes and a clean four-hour validation. Native Windows Claude's latest T0 evidence remains request received / approval pending, not confirmed ACK; native Codex T0 ACK is user-confirmed.
abruption commentedon Sep 22, 2026
#161 — privacy-filtered origin logging and bounded causal investigation
Outcome
Origin access logging is enabled and retained. We correlated one post-warmup public health timeout with a stalled, retransmitting specific KR-to-Cloudflare TCP connection, rather than merely a host-wide counter. We also reproduced three
no_authenticated_routefailures and isolated their immediate stage to the 8-second peer TLS handshake after successful admission, WebSocket upgrade, and receiver attach.These are narrower, evidence-backed findings, not a complete root-cause resolution of every #161 failure. The exact network fault location/owner and a causal link from the identified transport fault to each peer-TLS timeout remain unproven. #161 remains open; PR #162 is still draft and was not installed.
Controlled change and privacy checks
/var/log/caddy/session-peer-access.log,caddy:caddy,0600; 10MiB rollover, three rotated files and 72-hour rolling retention policy. Rotation retention is evaluated on rotation; it is not a promise that an inactive file is deleted exactly at 72 hours./var/backups/session-peer-161-20260922. Before SHA25622b15a0a9d8e5175793b1f16b5ead57d04d44390f7abe29e7f44e53b03baa171; applied SHA256b53aeed1b0a991a328285da2d2f0b3c9807fcc4a620f21738c2a253f40b10b83.Observation boundary and results
The existing RC2 campaign was retained as requested. Its post-change evidence is diagnostic, not a clean unchanged release soak. Boundary is also recorded in #150. The possible reload/WebSocket effect window ended at 17:39:13 KST.
Additional observation ran 17:37:53–17:46:23 KST, including warmup. It was stopped early after the four-route comparison and correlated failure evidence, rather than extending the investigation without establishing the remaining cause.
Post-warmup health samples:
One additional Mac mini health timeout occurred during warmup and is excluded from causal acceptance. Six authenticated probe-only runs used installed
1.0.0rc2; the runtime app.py hash matched published RC2 (de85f1852e1e742d54c977d58fa281319be0a36f3cc92823c635875fcd3a7a6f) on every run. No agent messages or resends were performed. Results: US→mini 2/2 passed, mini→book 1/2 passed, book→mini 0/1, mini→Windows/WSL 0/1. Collection errors: zero.Correlated public health failure
Diagnostic UUID
56d17532-093c-48cc-9ba2-4d0a6302da19, origin CF-Raya3f004d6afae08f9-HKG:bytes_acked=9002,bytes_retrans=9690, retransmission count 27, and an empty send queue.bytes_acked=9002, 139 bytes in its send queue,bytes_retrans=10277, retransmission count 32, and RTO 7520ms. ACK progress resumed in the subsequent sample.This establishes an origin-to-edge transport impairment associated with the health failure. It does not identify whether data or ACK loss/delay originates in OCI, an intermediate route, or Cloudflare, nor exclude additional delays on the client side. No claim about a particular provider's fault is made.
Authenticated Relay failures
All three failed probes returned successful control challenge/admission and Relay session responses, received WebSocket 101, and observed
relayAttached:true. Their peer-TLS handshake then raisedTimeoutErrorafter approximately 8.002s, producing the existingno_authenticated_routeresult.ea02e81c-d0f6-438b-a005-c8cb0226c939a3f002df7a5c0b1f-NRTf038b7ed-3dfa-493b-83e6-8134014592bba3f00694aa6d0a04-HKGa1a08435-ed6f-4233-adfc-f76909f23fcaa3f00956b822e6ba-HKGMatching origin WebSocket records last about 8.27–8.46 seconds and show 101, rather than HTTP authentication refusal. Admission/attach success rules out those preceding stages for these three probes. The payload-free origin records do not identify the opposite receiver stream strongly enough to attribute every stalled TLS handshake to the TCP impairment above. That remaining causal link is explicitly unproven. Do not extend the timeout or add retries and call that a root-cause fix.
Final state / evidence
/tmp/session-peer-161-observe.c2JnIP; manifest includes hashes of the frozen client/server metadata and collector sources. Sanitized logs remain on KR; backup/full configuration never left KR.92dc80a90872d85eda1e6453961ac0cbeadac3b916242d4662564e194d5eaede.8afcaebfaaa75d0745397b22709ff090cc4e264c320eb9e632a5351212474ecf.abruption commentedon Sep 22, 2026
Approved post-soak follow-up — prepared, not started
The user approved a maximum 15-minute Mac mini/MacBook receiver-side TLS/byte-count investigation after the existing four-hour RC2 campaign completes, followed by restoration of the original receiver execution method. Installed RC2, identities, policy, journals, security controls and timeouts must remain unchanged; no native messages or retries are included.
As of approximately 18:00 KST, the existing campaign and both original receivers are still running unchanged. Nominal T4 is 20:12:59 KST; the gate also requires the final network checkpoint and terminal completion, not just the wall clock. Aborted/incomplete runs do not authorize the transition.
The observational wrapper is prepared but not deployed. Five offline tests passed: byte/cookie passthrough without secret logging, cancellation without retry, and three completion-gate cases. A separate one-shot watcher is waiting for completion and will freshly discover this coordinator's Codex home before sending a continuation notification. The watcher itself does not restart receivers or perform the investigation; queued delivery will not be represented as consumed/completed work.
Prepared handoff:
/tmp/session-peer-161-followup.zQB9m5/HANDOFF.md, also linked from the RC2 campaign handoff. #161 remains open and PR #162 stays draft/uninstalled. Actual receiver instrumentation, paired-stream analysis and restoration are still pending the completion gate.abruption commentedon Sep 23, 2026
Independent Claude review — 2026-09-23 (correlation
ISSUE161-CLAUDE-20260923)The independent read-only investigation does not establish a root cause. Its strongest working hypothesis is intermittent transport stalls on the Cloudflare PoP↔KR origin path, amplified and obscured by receiver lifecycle and generic route errors. Provider/hop ownership and the causal link to individual peer-TLS failures remain unproven.
Reported RC2 soak evidence: direct 105/105, public Relay 78/140; all Relay failures were
no_authenticated_route, with no 35 s invoke timeout. Relay failures appeared from minute 0. A 30–64 min interval was much healthier with the same code, which argues against a deterministic code-only failure but does not exclude a timing-sensitive defect. The campaign ended after minute 172 withoutcomplete; it is incomplete, not a passing four-hour validation. Current Codex writer start time was 33 s after the last soak record, making a coordinator restart a plausible explanation for the missing terminal record, not for the preceding Relay failures. OS-level termination cause is unverified.The code behaviors were independently cross-checked at the published RC2 source: server receiver wait 60 s vs client wait 10 s; receiver
attach_timeout=65;relay_listenerresets backoff only on attach and otherwise rises to 5 s; errors collapse intono_authenticated_route. These are possible amplifiers, not an established explanation for each failed request.sessionsIssued - connectionsAcceptedandwaitingRoomsare useful aggregate signals, not exact per-request proof of where a particular request was lost. The prior three peer-TLS 8 s failures followed admission/101/attach, but the opposite receiver leg was not identified.Next work is split without declaring a retry-only fix: #163 investigates/bounds receiver lifecycle and stale attaches; #164 defines exact RC3 five-host, actual-ACK and clean four-hour validation. Draft PR #162 remains an uninstalled diagnostic/mitigation candidate and needs review against those findings. The
v1.0.0-rc.3milestone is preparation, not approval to publish or change production. Keep #161 open and stable v1.0.0 blocked until the mechanism and clean acceptance evidence are established.The earlier 15-minute receiver wrapper had a hard gate requiring an RC2
completerecord with elapsed >=14,400 s and T4 checkpoint. The run ended incomplete, so that authorization did not activate; any new live receiver swap/instrumentation needs fresh scope/approval. No new network test, message, infrastructure change or release is reported by this comment.fix(relay): diagnose route failures and safely retry setup timeouts (#…
fix(relay): shorten receiver reconnect gaps and count role_busy refusals
fix(relay): shorten receiver reconnect gaps and count role_busy refus…
release: prepare 1.0.0rc3 for Relay stability validation (#169)
abruption commentedon Sep 24, 2026
2026-09-24 stable decision: move this issue to v1.0.1 as monitoring, not as a claimed root-cause fix. RC4 mitigations (#162/#168) and the independent rc4-rerun-02 five-host 4-hour run had public health 241/241, probes 147/147, submissions 21/21 and no user-visible Relay transport failure. The earlier Cloudflare↔KR intermittent stall location/owner remains unproven, and one clean window cannot establish permanent absence. Keep this issue open; collect a correlated failure if it recurs. v1.0.0 notes explicitly disclose this boundary.
abruption commentedon Sep 24, 2026
Monitoring follow-up (not a root-cause fix): PR #196 adds UTC timestamps and per-attempt client→Relay→receiver metadata correlation; stacked PR #197 adds opt-in, per-leg opaque-frame ingress and completed socket-send counts/times. Both are designed to help attribute a future recurring failure without logging identities, credentials, or payloads. #197 is based on #196 and should be reviewed after it.
The RC2 Cloudflare↔KR stall location/owner and the causal link to a particular failed Relay attempt remain unproven. A completed Relay socket send does not prove receiver delivery. Keep #161 open for a correlated recurrence and causal investigation. The older unmerged #161 observability worktree was preserved unchanged.
abruption commentedon Sep 24, 2026
PR #197 adds monitoring metadata only; it does not establish the cause of the intermittent public Relay stall. GitHub automatically closed this issue when the merged PR body contained a negated closing phrase. The PR body has been corrected and this issue reopened. Continue v1.0.1 monitoring; the Cloudflare↔KR/OCI/transit fault location remains unconfirmed.
abruption commentedon Sep 27, 2026
The v1.0.2 release preparation PR #224 inadvertently triggered GitHub's automatic closing rule because its description contained a negated closing phrase. That phrase has been removed and this issue has been reopened.
v1.0.2 includes stability fixes and additional diagnostics, but the root cause and owner of the intermittent public Relay path stalls remain unconfirmed. Keep this issue in the monitoring milestone; the release does not claim a network root-cause fix.