[Bug]: Relay never re-verifies managed tunnel ingress when the recorded origin matches, so drift breaks T3 Connect permanently #27952

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

infra/relay (the form's list has no relay option). The host side is apps/server.

Summary

The relay records each managed endpoint's origin, the host and port pair, in the allocation table. It treats that record as proof that the Cloudflare tunnel serves the same origin. No repair path reads the tunnel's live configuration when the recorded origin matches what the host reports.

When the real ingress on Cloudflare drifts to a different port, the drift is permanent. The tunnel stays healthy and connected. Every request it proxies hits a dead local port. Clients see 502 and origin_unreachable forever, and the documented fix, restarting T3 Code, changes nothing, because the restart takes the same matching-origin path.

We hit this in production on 2026-10-30. The tunnel ingress served http://127.0.0.1:4711, the server listened on 3773, and the allocation recorded 127.0.0.1:3773.

This is the residual gap after #7458. That issue covered the case where the recorded origin differed from the server's port, and the fix added the origin-sync path that repairs it. Here the recorded origin is correct and the actual ingress is not, so the sync never runs.

Steps to reproduce

Minimal, as a test in infra/relay/src/environments/ManagedEndpointProvider.test.ts:

1. Give the allocation origin 127.0.0.1:3773, with readyAt and dnsRecordId set.
2. Make the Cloudflare mock report a tunnel whose configuration serves http://127.0.0.1:4711.
3. Call reconcileOrigin with origin 127.0.0.1:3773.
4. It returns ready and leaves the ingress untouched.

To reach this state on a real environment, one half-completed origin sync is enough. The server moves to a new port, putConfiguration writes that port to Cloudflare, and markReady then loses the generation claim. The allocation keeps the old port while the ingress serves the new one. When the server returns to the old port, every later registration matches the allocation and skips the repair. We believe this is how the production occurrence started, but we could not prove the historical write, so treat this paragraph as reconstruction. The permanent, unrepaired drift is the part we verified live, in the logs below.

Expected behavior

• After a boot registration or a health check, the recorded origin and the tunnel's actual ingress agree.
• When they disagree, the next reconcileOrigin call repairs the ingress.

Actual behavior

• reconcileOrigin returns ready on an origin match without reading the tunnel configuration (ManagedEndpointProvider.ts L780-L785).
• reuseReadyTunnel reuses the tunnel and skips ingress and DNS repair for a matching origin (L647-L651).
• The only routine writer of the ingress is a full provision, and a linked environment that reports the recorded origin never triggers one.

Impact

Blocks work completely. Remote access to the environment stays dead until someone toggles T3 Connect off and on by hand.

Version or commit

Relay production at https://relay.t3.codes, serving host 0.1.6-nightly.20261030.3940. Code links point at 7471a6c2093e7c04851ba06d282b2d709da72d6c on main.

Environment

macOS 27 arm64. T3 Code Nightly desktop 0.1.6-nightly.20261030.3940. cloudflared 2026.10.4, managed by the server. The environment was linked through the desktop UI (Settings, Sharing and access), not the CLI.

Logs or stack traces

Client diagnostics, verbatim:

This device: ok
T3 Connect: ok
Aryan's MacBook Air tunnel: ok
T3 Code on Aryan's MacBook Air: FAILED

Cause: origin_unreachable
HTTP status: 502
Attempts: 3
Route: T3 Connect
App version: 2.2.0
Host version: 0.1.6-nightly.20261030.3940
Error: Relay could not reach the environment endpoint (endpoint_request_failed).
Trace ID: febb511cdc0a14975a2005b275be8c7e

Host-side evidence, collected while the failure was live. The environment ID is e4465974-9b0f-4a39-a8b7-f2c5006f6ebb and the endpoint hostname is prod-6dce4976cc01ffc7.t3coderelay.com.

cloudflared's remote config, read through its metrics port, shows the ingress pointing at the wrong port while the connector is fully healthy:

$ curl -s http://127.0.0.1:51827/config | jq '.config.ingress[0]'
{
  "hostname": "prod-6dce4976cc01ffc7.t3coderelay.com",
  "service": "http://127.0.0.1:4711",
  ...
}
$ curl -s http://127.0.0.1:51827/ready
{"status":200,"readyConnections":4,"connectorId":"ae387feb-8433-47d2-98af-c37513794561"}

The public endpoint returns 502, the recorded port answers 200, and the ingress port refuses connections:

$ curl -s -o /dev/null -w '%{http_code}\n' https://prod-6dce4976cc01ffc7.t3coderelay.com/.well-known/t3/environment
502
$ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3773/.well-known/t3/environment
200
$ curl -s http://127.0.0.1:4711/
curl: (7) Failed to connect to 127.0.0.1 port 4711

The host's own state was self-consistent. ~/.t3/userdata/server-runtime.json reported origin http://127.0.0.1:3773, and the relay-confirmed origin marker in the secret store recorded 127.0.0.1:3773 as well. Boot registrations succeeded on every start, which means the relay's allocation also recorded 127.0.0.1:3773. A differing record would have triggered the repair path described below.

Root cause

All links are to 7471a6c2093e7c04851ba06d282b2d709da72d6c on main.

1. reconcileOrigin runs on every boot registration. When allocation.origin equals the origin the host reports, it returns ready immediately (L780-L785). It verifies that the tunnel still exists on Cloudflare (L759-L779), but it never reads the tunnel's configuration.
2. The repair path runs only when the two origins differ (L787-L846). It writes the new ingress with putConfiguration and then persists the origin with markReady, guarded by WHERE generation = input.generation (ManagedEndpointAllocations.ts L369-L375). If markReady loses that race, reconcileOrigin fails with claim-lost (L839-L846). Cloudflare then serves a port the allocation does not record.
3. recoverManagedEndpoint calls provision with reuseReadyTunnel: true. For a matching origin, reuseReadyTunnel returns the tunnel "with only a liveness check and a fresh token, skipping ingress and DNS repair" (L156-L160, L637-L651). The comment states the tunnel "already serves this origin", but nothing verifies that.
4. A full provision rewrites the ingress (L1315-L1374). A linked environment that reports the recorded origin never triggers one.
5. The relay maps cloudflared's 502 and 503 to origin_unreachable (endpointFailure.ts L24-L30). The host's reachability check then tells the user to "Restart T3 Code so the tunnel uses the port the server listens on" (ConnectReachability.ts L110-L116). A restart re-runs step 1 and changes nothing. The troubleshooting doc gives the same advice for this failure mode.

Suggested fixes

1. In reconcileOrigin, when the recorded origin matches, also read the tunnel's live configuration and compare the ingress service with formatOriginService(input.origin). On mismatch, run the existing putConfiguration and markReady path. The registration already pays for one Cloudflare read, the tunnels.get existence check, so one more read is cheap. This turns every boot registration into a convergence point.
2. Optionally, on the host side: inspectConnector already reads cloudflared's /config (ManagedEndpointRuntime.ts L740-L762). When the ingress service differs from the server's own origin, the reachability report could name both ports, or the host could request a relink instead of telling the user to restart.

Workaround

On the host, open Settings, go to Sharing and access, and turn Enable T3 Connect off. Wait until the tunnel count drops, then wait about 30 seconds, and turn it on. The relink runs a full provision, which rewrites the ingress. The endpoint returned 200 right after, and the client check passed.

One caveat: the first relink attempt, immediately after the off toggle, failed at stage configure-tunnel with trace ID fb747b2da0ab9e340e40b7dc77d8c068. It looked like a Cloudflare-side race on the just-released tunnel. A retry 30 seconds later succeeded.

Possibly related: #8844 reports the same client symptom, endpoint_request_failed, with the cause undiagnosed. I am not claiming it is this bug.