From a8d14af18d88a00ab37b8ca6b2fdc8097a315aa4 Mon Sep 17 00:00:00 2001 From: Julius Marminge <51714798+juliusmarminge@users.noreply.github.com> Date: Thu, 24 Sep 2026 21:30:55 -0700 Subject: [PATCH 1/4] docs(relay): add the legacy tunnel cleanup rollout runbook Order the client release before enabling legacy cleanup, point the dry-run at Cloudflare's totals rather than a single sweep's sample, give the Axiom query for the sweep counters, and list when to turn the switch back off. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/operations/release.md | 22 +++++++++++++++++++++- 1 file changed, 21 insertions(+), 1 deletion(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 4185f87b2a75..3d64ca56a626 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -202,11 +202,17 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the - On an older build linked from web or mobile, the host stays offline until T3 Code on that computer is updated. +Ship the web and mobile builds that show the offline reason before enabling legacy cleanup, so a +user whose host is affected sees what to do. The relay adds the `tunnel_released_at` allocation +column in its first deploy with this change; the legacy switch stays `off` until you set it. + 1. Run `vp run --filter t3code-relay tunnels:census` with a read-only Cloudflare token. It counts tunnels in every relay stage. The reaper only sees its own stage's tunnels, so clean up the rest by hand. 2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`, - `totalDown`, and `totalInactive` on the sweep spans for a day. + `totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the + tunnels a sweep inspected, at most 500 per status, so use `totalDown` and `totalInactive` for the + size of the backlog. 3. Run the legacy steps of the disposable-host canary below. 4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer" message are live. Without them, a user whose older host lost its tunnel only sees it as offline. @@ -217,6 +223,20 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the budget ran out or Cloudflare rate-limited a deletion. The counters don't say which; the relay logs a warning with the Cloudflare error for each failed deletion. +In Axiom, filter the relay traces dataset on `name == "relay.managed_endpoint_reaper.sweep"` and +chart the `attributes.custom.relay.managed_endpoint_reaper.*` fields over time. + +Set the legacy mode back to `off` and deploy if any of these happen: + +- `failed` stays above a few per sweep. Read the warning log for the Cloudflare error. +- Users report an environment that is offline with the update message after they have updated T3 + Code on that computer and restarted it. +- Relay request errors rise while sweeps run. Deletions share the Postgres connection pool with + request handlers. + +Turning the switch off stops new deletions. Deleted tunnels stay deleted; their hosts recover as +described above. + ### Disposable-host canary This test has not been run against a real Cloudflare account. Run it against a disposable relay From b57cb11f7a18f45fa3e0a19bd8298d80bec11b6f Mon Sep 17 00:00:00 2001 From: Julius Marminge <51714798+juliusmarminge@users.noreply.github.com> Date: Thu, 24 Sep 2026 22:42:22 -0700 Subject: [PATCH 2/4] docs(relay): treat Cloudflare's idle totals as an upper bound on the backlog Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/operations/release.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 3d64ca56a626..598d6ba2017d 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -211,8 +211,10 @@ column in its first deploy with this change; the legacy switch stays `off` until by hand. 2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`, `totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the - tunnels a sweep inspected, at most 500 per status, so use `totalDown` and `totalInactive` for the - size of the backlog. + tunnels a sweep inspected, at most 500 per status. `totalDown` and `totalInactive` are Cloudflare's + counts of all idle tunnels in this stage, including ones the reaper skips, so they are an upper + bound on the backlog. The share of `wouldDeleteLegacy` in each sweep's `scanned` estimates how + much of that total is eligible. 3. Run the legacy steps of the disposable-host canary below. 4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer" message are live. Without them, a user whose older host lost its tunnel only sees it as offline. From eafb488fcdc54c65091b50e488a6a8231fb57898 Mon Sep 17 00:00:00 2001 From: Julius Marminge <51714798+juliusmarminge@users.noreply.github.com> Date: Thu, 24 Sep 2026 23:22:30 -0700 Subject: [PATCH 3/4] docs(relay): scope Cloudflare's totals to tunnels past the grace periods Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/operations/release.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index 598d6ba2017d..c502abe48cc3 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -212,8 +212,8 @@ column in its first deploy with this change; the legacy switch stays `off` until 2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`, `totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the tunnels a sweep inspected, at most 500 per status. `totalDown` and `totalInactive` are Cloudflare's - counts of all idle tunnels in this stage, including ones the reaper skips, so they are an upper - bound on the backlog. The share of `wouldDeleteLegacy` in each sweep's `scanned` estimates how + counts of this stage's tunnels down for over five minutes and never connected for over an hour. + They include ones the reaper skips, so they are an upper bound on the backlog. The share of `wouldDeleteLegacy` in each sweep's `scanned` estimates how much of that total is eligible. 3. Run the legacy steps of the disposable-host canary below. 4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer" From d3367bef3b800c56eefb515e762300069051492d Mon Sep 17 00:00:00 2001 From: Julius Marminge <51714798+juliusmarminge@users.noreply.github.com> Date: Mon, 5 Oct 2026 18:16:55 -0700 Subject: [PATCH 4/4] docs(relay): say the legacy switch stops only legacy deletions Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/operations/release.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/operations/release.md b/docs/operations/release.md index c502abe48cc3..baff2ed56b23 100644 --- a/docs/operations/release.md +++ b/docs/operations/release.md @@ -236,8 +236,9 @@ Set the legacy mode back to `off` and deploy if any of these happen: - Relay request errors rise while sweeps run. Deletions share the Postgres connection pool with request handlers. -Turning the switch off stops new deletions. Deleted tunnels stay deleted; their hosts recover as -described above. +Turning the legacy mode off stops new legacy deletions; `RELAY_TUNNEL_CLEANUP_MODE` keeps deleting +tunnels of hosts with recovery while it is `enabled`. Deleted tunnels stay deleted; their hosts +recover as described above. ### Disposable-host canary