Skip to content

infra(relay): add c4 and c5 to the committed fence set - #21166

Closed
Jinwoo-H wants to merge 1 commit into
mainfrom
relay-fence-c4-c5
Closed

Jinwoo-H wants to merge 1 commit into
mainfrom
relay-fence-c4-c5

Conversation

@Jinwoo-H

@Jinwoo-H Jinwoo-H commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​7 $\color{#cf222e}{\Huge{\mathbf{−}}}$​6 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​1
Prod 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​14 $\color{#cf222e}{\Huge{\mathbf{−}}}$​1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​13

ELI5

Two of our servers have been sitting almost empty for weeks, each still holding a machine and ten database connections. This adds them to the list of servers approved for switch-off.

It does not switch them off. It is the paperwork that a separate, guarded procedure checks before it pulls the plug.

What Changed

One edit to cloud/infra/terraform/environments/production.tfvars: production-gce-c4 and production-gce-c5 join relay_gce_fenced_cells. The list is reflowed to one entry per line so the two additions read as two added lines. The budget test's cell figure drops accordingly.

Merge this immediately before running fence-source for c4 and then c5, and nothing else. That is the whole intent of the PR.

relay-gce-cells.tf derives each cell's MIG target size directly from this list, and the lifecycle block that used to ignore operational target_size drift is gone. So an entry sitting on main is not a note about future intent. It is a standing instruction that any Terraform apply reaching that cell may carry out, without the drain and attestation the fence workflow exists to produce. The shorter that entry's life on main, the smaller the window.

The guarded path is Deploy Relay Production Multi-Target in fence-source mode. It confirms the committed fence set at step 2 of its prepare-and-guard sequence, then reads live topology and runs the admission, assignment, lease, migration, connection, heartbeat, backend, and route guards. It refuses a cell whose admission is still enabled or that owns any work, resizes the instance group to zero, proves no instance remains and the exact-incarnation heartbeat is stale, and only then records an attestation.

Before and after, as a user experiences it: nothing. Nobody is meaningfully connected to c4 or c5. After the fence workflow runs, two virtual machines are gone and 20 Postgres connections return.

Why

c4 and c5 are existing-only. RelayCellAdmissionSelector refuses to move an existing-only cell back to migration-only or general once the selector generation is past zero, so they are retired capacity that cannot be reclaimed. They were measured on 2026-09-16 carrying about one connection each on images 40 days old. Six of the eight existing-only cells are already in this list; these two complete it.

Why a separate PR rather than folding it into the asia-pool change (#21163): the two have nothing to do with each other, and merging them together would put these entries on main for however long that unrelated change takes to land. Keeping the fence entries in a PR whose only job is to precede the fence run keeps each apply's blast radius explicit and reviewable.

Why not also c17 and c18, which are equally idle: they are migration-only. cloud/docs/orca-relay-operations.md requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector compare-and-swap before it can be fenced, and cloud/dev/scripts/deploy-relay-gce-multi-target.mjs enforces it. Terraform cannot express that step, it is irreversible, and c17/c18 are the only two cells left that could still be promoted to general. Retiring them is a capacity decision that should be made on its own.

Linked Issue

Fixes #

Visual Proof

N/A — infrastructure configuration. There is no user interface surface.

Testing

Run from cloud/, with Terraform 1.15.8 locally to match the terraform_version: 1.15.8 pin in the workflows.

Gate Result
terraform fmt -check -recursive pass
terraform init -backend=false then terraform validate pass
pnpm -r test 1044 pass, 115 skipped, 0 fail
Main node --test script list 596 pass, 0 fail
pretest script list 156 pass, 0 fail

No Terraform apply was run and no remote state was touched. init used -backend=false.

The budget test's cell figure is pinned at 210, which is main's 230 less 20 for these two cells, giving 365 operating against a usable ceiling of 390. That figure assumes this PR lands on main as it stands today. If #21163 lands first the correct figure is 228, because it adds 18 for the three asia-east2 pools at 16. Whichever of the two lands second needs its pinned figure reconciled; the conflict is one line in one file.

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below

AI Disclosure

Review

What a reviewer should confirm before approving

The live admission state of c4 and c5 is runtime state in the director's database. It is not checked into this repository and cannot be read from these files. The claim that both are existing-only rests on a selector inspection taken on 2026-09-16, and on the corroborating fact that six of the eight existing-only cells are already in this list. Re-confirm it with Operate Relay Asia Admission in inspect mode before merging, since merging is meant to be the last step before the fence run.

If either cell turns out to be migration-only rather than existing-only, this PR is wrong for that cell and it needs retire-migration-cell first.

A stale line in the runbook

cloud/docs/orca-relay-operations.md still says Terraform "intentionally ignores only operational MIG target_size drift, so a later apply cannot recreate a fenced source". That stopped being true when Terraform took ownership of the fence and the lifecycle ignore was removed. It is the exact sentence that would talk a reader out of the caution this PR is built around. It should be corrected separately; this PR does not touch documentation.

Related

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Security: no identity, permission, or trust boundary changes. Cross-platform: Terraform and a Node test file, no platform-dependent behavior. Remote SSH and Mobile: unaffected. Backwards compatibility: no wire format, API, or client contract changes. Performance: returns 20 Postgres connections and two virtual machines once the fence workflow runs; no effect on serving cells.

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

Both are existing-only cells the admission selector can never place on again.
They measured ~1 connection each on 40-day-old images on 2026-09-16 while each
still holds a VM and 10 Postgres connections.

Merge this immediately before running fence-source for c4 and then c5, and
nothing else. Terraform reads a cell's MIG target size straight from this list
(relay-gce-cells.tf) with no lifecycle ignore, so an entry sitting on main ahead
of its drain is a standing instruction that any apply reaching that cell may
carry out without the attestation the workflow exists to produce.

The budget test drops 20 connections, 230 -> 210, leaving 365 operating against
a usable ceiling of 390. Merged after the asia-pool change the figure is 228.
@OrcaWin

OrcaWin commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Closing this public-repository PR because #31792 (307f0cd) moved production Terraform, the committed fence set, and operator budget contracts into the private stablyai/orca-cloud repository.

I reviewed both changed files and the operational intent. The production tfvars and the connection-budget operator test no longer exist in Orca main, and cloud/README.md explicitly makes the private operations repo the owner of Terraform and deployment configuration. Adding c4/c5 to a resurrected public tfvars file would not update the configuration the guarded fence procedure now consumes. Any remaining c4/c5 retirement decision must be evaluated against the current private fence set and live admission state. This closure does not assert those cells were fenced; this account cannot inspect that private repository or live topology.

@OrcaWin OrcaWin closed this Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants