Repository navigation
Conversation
Both are existing-only cells the admission selector can never place on again. They measured ~1 connection each on 40-day-old images on 2026-09-16 while each still holds a VM and 10 Postgres connections. Merge this immediately before running fence-source for c4 and then c5, and nothing else. Terraform reads a cell's MIG target size straight from this list (relay-gce-cells.tf) with no lifecycle ignore, so an entry sitting on main ahead of its drain is a standing instruction that any apply reaching that cell may carry out without the attestation the workflow exists to produce. The budget test drops 20 connections, 230 -> 210, leaving 365 operating against a usable ceiling of 390. Merged after the asia-pool change the figure is 228.
|
Closing this public-repository PR because #31792 (307f0cd) moved production Terraform, the committed fence set, and operator budget contracts into the private I reviewed both changed files and the operational intent. The production tfvars and the connection-budget operator test no longer exist in Orca main, and |
ELI5
Two of our servers have been sitting almost empty for weeks, each still holding a machine and ten database connections. This adds them to the list of servers approved for switch-off.
It does not switch them off. It is the paperwork that a separate, guarded procedure checks before it pulls the plug.
What Changed
One edit to
cloud/infra/terraform/environments/production.tfvars:production-gce-c4andproduction-gce-c5joinrelay_gce_fenced_cells. The list is reflowed to one entry per line so the two additions read as two added lines. The budget test's cell figure drops accordingly.Merge this immediately before running
fence-sourcefor c4 and then c5, and nothing else. That is the whole intent of the PR.relay-gce-cells.tfderives each cell's MIG target size directly from this list, and thelifecycleblock that used to ignore operationaltarget_sizedrift is gone. So an entry sitting onmainis not a note about future intent. It is a standing instruction that any Terraform apply reaching that cell may carry out, without the drain and attestation the fence workflow exists to produce. The shorter that entry's life onmain, the smaller the window.The guarded path is
Deploy Relay Production Multi-Targetinfence-sourcemode. It confirms the committed fence set at step 2 of its prepare-and-guard sequence, then reads live topology and runs the admission, assignment, lease, migration, connection, heartbeat, backend, and route guards. It refuses a cell whose admission is still enabled or that owns any work, resizes the instance group to zero, proves no instance remains and the exact-incarnation heartbeat is stale, and only then records an attestation.Before and after, as a user experiences it: nothing. Nobody is meaningfully connected to c4 or c5. After the fence workflow runs, two virtual machines are gone and 20 Postgres connections return.
Why
c4 and c5 are existing-only.
RelayCellAdmissionSelectorrefuses to move an existing-only cell back to migration-only or general once the selector generation is past zero, so they are retired capacity that cannot be reclaimed. They were measured on 2026-09-16 carrying about one connection each on images 40 days old. Six of the eight existing-only cells are already in this list; these two complete it.Why a separate PR rather than folding it into the asia-pool change (#21163): the two have nothing to do with each other, and merging them together would put these entries on
mainfor however long that unrelated change takes to land. Keeping the fence entries in a PR whose only job is to precede the fence run keeps each apply's blast radius explicit and reviewable.Why not also c17 and c18, which are equally idle: they are migration-only.
cloud/docs/orca-relay-operations.mdrequiresretire-migration-cellto move a migration-only cell to existing-only through a generation-bound selector compare-and-swap before it can be fenced, andcloud/dev/scripts/deploy-relay-gce-multi-target.mjsenforces it. Terraform cannot express that step, it is irreversible, and c17/c18 are the only two cells left that could still be promoted to general. Retiring them is a capacity decision that should be made on its own.Linked Issue
Fixes #
Visual Proof
N/A— infrastructure configuration. There is no user interface surface.Testing
Run from
cloud/, with Terraform 1.15.8 locally to match theterraform_version: 1.15.8pin in the workflows.terraform fmt -check -recursiveterraform init -backend=falsethenterraform validatepnpm -r testnode --testscript listpretestscript listNo Terraform apply was run and no remote state was touched.
initused-backend=false.The budget test's cell figure is pinned at 210, which is
main's 230 less 20 for these two cells, giving 365 operating against a usable ceiling of 390. That figure assumes this PR lands onmainas it stands today. If #21163 lands first the correct figure is 228, because it adds 18 for the three asia-east2 pools at 16. Whichever of the two lands second needs its pinned figure reconciled; the conflict is one line in one file.AI Disclosure
Review
What a reviewer should confirm before approving
The live admission state of c4 and c5 is runtime state in the director's database. It is not checked into this repository and cannot be read from these files. The claim that both are existing-only rests on a selector inspection taken on 2026-09-16, and on the corroborating fact that six of the eight existing-only cells are already in this list. Re-confirm it with
Operate Relay Asia Admissionininspectmode before merging, since merging is meant to be the last step before the fence run.If either cell turns out to be migration-only rather than existing-only, this PR is wrong for that cell and it needs
retire-migration-cellfirst.A stale line in the runbook
cloud/docs/orca-relay-operations.mdstill says Terraform "intentionally ignores only operational MIGtarget_sizedrift, so a later apply cannot recreate a fenced source". That stopped being true when Terraform took ownership of the fence and the lifecycle ignore was removed. It is the exact sentence that would talk a reader out of the caution this PR is built around. It should be corrected separately; this PR does not touch documentation.Related
Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
Security: no identity, permission, or trust boundary changes. Cross-platform: Terraform and a Node test file, no platform-dependent behavior. Remote SSH and Mobile: unaffected. Backwards compatibility: no wire format, API, or client contract changes. Performance: returns 20 Postgres connections and two virtual machines once the fence workflow runs; no effect on serving cells.
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)