Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 24 additions & 1 deletion docs/operations/release.md
Original file line number Diff line number Diff line change
Expand Up @@ -202,11 +202,19 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the
- On an older build linked from web or mobile, the host stays offline until T3 Code on that computer
is updated.

Ship the web and mobile builds that show the offline reason before enabling legacy cleanup, so a
user whose host is affected sees what to do. The relay adds the `tunnel_released_at` allocation
column in its first deploy with this change; the legacy switch stays `off` until you set it.

1. Run `vp run --filter t3code-relay tunnels:census` with a read-only Cloudflare token. It counts
tunnels in every relay stage. The reaper only sees its own stage's tunnels, so clean up the rest
by hand.
2. Set the legacy mode to `dry-run`, deploy, and read `wouldDeleteLegacy`, `legacyOver30Days`,
`totalDown`, and `totalInactive` on the sweep spans for a day.
`totalDown`, and `totalInactive` on the sweep spans for a day. `wouldDeleteLegacy` counts only the
tunnels a sweep inspected, at most 500 per status. `totalDown` and `totalInactive` are Cloudflare's
counts of this stage's tunnels down for over five minutes and never connected for over an hour.
They include ones the reaper skips, so they are an upper bound on the backlog. The share of `wouldDeleteLegacy` in each sweep's `scanned` estimates how
much of that total is eligible.
3. Run the legacy steps of the disposable-host canary below.
4. Before enabling, confirm the web and mobile builds that show the "update T3 Code on that computer"
message are live. Without them, a user whose older host lost its tunnel only sees it as offline.
Expand All @@ -217,6 +225,21 @@ A deleted legacy tunnel keeps its allocation, so its hostname is kept. When the
budget ran out or Cloudflare rate-limited a deletion. The counters don't say which; the relay
logs a warning with the Cloudflare error for each failed deletion.

In Axiom, filter the relay traces dataset on `name == "relay.managed_endpoint_reaper.sweep"` and
chart the `attributes.custom.relay.managed_endpoint_reaper.*` fields over time.

Set the legacy mode back to `off` and deploy if any of these happen:

- `failed` stays above a few per sweep. Read the warning log for the Cloudflare error.
- Users report an environment that is offline with the update message after they have updated T3
Code on that computer and restarted it.
- Relay request errors rise while sweeps run. Deletions share the Postgres connection pool with
request handlers.

Turning the legacy mode off stops new legacy deletions; `RELAY_TUNNEL_CLEANUP_MODE` keeps deleting
tunnels of hosts with recovery while it is `enabled`. Deleted tunnels stay deleted; their hosts
recover as described above.

### Disposable-host canary

This test has not been run against a real Cloudflare account. Run it against a disposable relay
Expand Down
Loading