[Jinwoo-H]

	Files	Added	Deleted	Net
Test	15	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​762	$\color{#cf222e}{\Huge{\mathbf{−}}}$​55	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​707
Prod	17	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​396	$\color{#cf222e}{\Huge{\mathbf{−}}}$​114	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​282

ELI5

Hong Kong has three relay servers (c27, c28, c29) for users in Asia, and they are close to full. This PR writes down a fourth one, c30, built exactly like the other three. It also teaches every script and workflow that lists the Asia servers about c30. Merging it creates nothing. Every production step comes later, as its own separately approved dispatch.

What Changed

Before: operators could not add a fourth Asia cell. Every Asia tool only accepted exactly C27, C28, and C29. The topology workflow would also have rejected the committed C27-C29 entries themselves. Its validators still expected the old database pool of 10, while the committed cells have run at 16 since #21163.

After: production-gce-c30 is declared, disabled, and every Asia tool accepts it as its own additive step. The C30 steps never change, register, or re-promote the live launch cells. C30 is promoted the same way C27 was: a five-minute production load test on C30 itself, with an automatic return to migration-only if it fails.

The new cell entry copies c27 field for field. It uses asia-east2, e2-standard-4, the same 30 GB boot disk and boot image, 6,000 request units, and a pool of 16 with the same comment. It pins the relay image the production director serves, initially_enabled = false, and 3,000/60 connection limits. C30 is also added to the regional rehome source list, like the other Asia cells. That setting is per cell and changes no other cell's template. C27-C29 keep their committed image lines, which this PR does not touch.

Live digests, read on 2026-09-22 from production. Re-read both before the topology plan. If the director has moved, repin C30 again first.

What	Digest
Production director, Cloud Run revision 00664	sha256:4158d8a2e18e9caec439d257f0c1e45d92ffea8c0262f057b2f08c76a134bcf0
Live C27, C28 and C29 instance templates, dated 2026-09-19	sha256:17ac913b96bb55b9d2d3302ff178556e3e6e5fbc1d9a98283d0f6f345807d575
Committed C27-C29 image in the file, stale	sha256:5aedbca5c86de24c8b4d4bf7e3b444b76c712f281ede916cb9d90f70cad1e563

C30 is pinned to the director digest because promotion requires the director and C30 to report the same digest. Every Asia check compares only the cells named in that run, so C30 on the director digest and C27-C29 on theirs pass side by side. Tests cover this for the topology input, the director cell list, and promotion. The topology plan does not use the committed C27-C29 images. It reads their served images from Terraform state at run time.

Zone: asia-east2-a. No doc or resource says how to pick an Asia zone. Production cells rotate a, b, c by ordinal: c25 is c, c26 is a, and c27/c28/c29 are a/b/c. Continuing that rotation gives c30 zone a. It shares a zone with c27, which is unavoidable with four cells in three zones.

How the topology plan stays limited to C30. The workflow targets C30's template, VM group, and backend, plus the shared routing map. The routing map points at every cell's backend, and Terraform plans everything a target depends on. So every live cell's backend, VM group, and template enters the plan. The committed image in tfvars trails what the same-cap rollouts actually deployed, so without help the plan would replace every live cell. The workflow now reads each non-target cell's deployed image out of its template in Terraform state. It plans that cell at the deployed image, which leaves it unchanged. The state read keeps only template startup scripts, so no secret value reaches disk or logs. The plan validator now names any change to a cell outside the wave and rejects it, including replacements and deletes.

How C30's canary proves C30. Placement sends a new host to the least-loaded general cell in its preferred region. An empty C30 beside three nearly full cells therefore receives the canary's control, because each run uses a fresh random host identity with no prior assignment. The load report now records every cell origin its controls connected to. C30's evidence must show exactly https://c30.relay.onorca.dev, must read C30's own runtime metrics, and is bound to the selector generation the promotion produced. C27's evidence now carries the same placement check.

Each enumeration site and the decision for it:

Site	Decision
production.tfvars cell map and rehome source list	Add. These declare the cells that exist, and every Asia cell carries rehome trust.
validate-relay-asia-topology-plan.mjs and prepare-relay-asia-topology-input.mjs	Add C30 as a second reviewed wave beside the launch trio. A request must equal exactly one wave. Mixed or four-cell sets are rejected. The pool pin is now 16 for production and 10 for staging. Any change to a cell outside the wave is rejected by name.
cloud-deploy-relay-asia-topology.yml	Add a production:production-gce-c30 case. The plan and the post-apply readback both use the deployed-image overlay described above.
relay-live-cell-image-overlay.mjs (new)	Builds that overlay. It fails if a non-target cell has no template in state, if a template does not carry exactly one pinned relay image, or if either input is empty. A target cell with no template yet, which is C30 today, is kept at its committed image.
prepare-relay-asia-director-cells.mjs	No list here. The pool check changes from a flat 10 to 16 for production and 10 for staging. Without that it rejects C30's real state output.
operate-relay-asia-admission.mjs	C30 joins the set of Asia cells that exist. Registration waves are the trio or C30 alone. Promotion waves are C27, then C28+C29, then C30. C30 promotion requires C27, C28, and C29 to be general. Rollback accepts a reviewed wave or the whole set, and rejects mixed sets such as C27+C30.
cloud-operate-relay-asia-admission.yml	The timed canary, its evidence upload, and its automatic rollback now run for C27 or C30. C30 promotion takes no input evidence, because its own canary is the proof. C28/C29 still require C27's canary evidence under its unchanged artifact name.
relay-asia-rollout-evidence.mjs and load-relay-controls.mjs	The canary evidence builder takes the canary cell, C27 or C30, and checks the placed cell origin.
cloud-deploy-relay-production-same-cap-job.yml case list	Add c30 to the 3,000-cap / asia-east2 / pool 16 branch, so C30 can be rolled later like its siblings.
relay-production-same-cap-wave.mjs	Add C30 to the migration-only cells, beside C17 and C18, until promotion. A same-cap roll then hands C30 back isolated. If C30 were listed as general, a same-cap rollback on it would switch it to general and skip its canary. This listing also leaves a same-cap roll as the way to re-image C30 if the director moves to a new image before promotion.
probe-relay-rehome-trust.mjs allow-list	Add. Without it, C30's first same-cap roll would fail after the apply and leave C30 serving but isolated. That happened to C27 on 2026-09-07.
relay-same-cap-shadow-gate-verdict.mjs fleet pool list	Not added yet. A listed cell with no metrics reads as unverified, which would turn every verdict into a warning until C30 serves. Adding C30 belongs to the change after its promotion.
outputs.tf relay_gce_cell_deployments	Six per-cell lookups of the VM group, backend and template now fall back to null. Before this, once C30 was in the file but not yet created, every production terraform console call printed a warning into its output. That broke the director deploy, same-cap and capacity jobs until C30's topology apply. Now C30 reads with null VM group, backend and template fields until then.
Tests that used production-gce-c30 as an example of an unknown cell	Moved to production-gce-c31, and C30 was added to the matching accepted lists.
relay-ops environment-config.test.ts	No change. It uses a synthetic fixture, and the real inventory is read from tfvars, so it now includes C30.
incident-monitor.ts	No change. It records history.
relay-cloud-sql-connection-budget.test.mjs	Updated. It pins the fleet numbers on purpose. A new test covers the refusal of a C30 at pool 10 beside three cells at 16.
cloud/docs/relay-workflows.md, cloud/infra/terraform/README.md	The launch-history sentences stay. New sentences say the Asia set is C27-C30 and describe the C30 wave, its plan, and its canary.

Why

Why add one cell instead of raising the cap on the existing cells. How fast a cell can take back reconnecting hosts is a per-cell limit, so a bigger cap does not help a reconnect storm. The cap is also built into each cell's VM template. Raising it means rebuilding every Asia cell, which drains its hosts during the rebuild. A new cell at the already-reviewed shape reuses every existing proof. It adds 3,000 connections and 6,000 request units without touching a live cell.

Why C30 is its own wave instead of a new four-cell set. A four-cell plan would ask Terraform to create or re-review all four cells. With one wave per addition, C30's plan creates only C30's resources and one routing update, and every live cell must read as unchanged.

Why plan live cells at their deployed image instead of first correcting the committed images. A correction would be a separate reviewed change that has to track every same-cap rollout. The overlay reads the deployed truth at plan time, the same way the same-cap job already writes its target image into a runtime variable file. Any other drift still fails the plan, so the overlay cannot hide a real change.

Why C30 gets its own production canary instead of staging evidence. Staging evidence proves the image on a staging cell, not C30 under production traffic. C28 and C29 were promoted only after a production canary on C27, and C27 had an automatic rollback. C30 now gets the same bar: a real production load that lands on C30, C30's own metrics, generation binding, and an automatic return to migration-only on failure. It costs about six minutes.

Connection budget

The budget script is node cloud/dev/scripts/relay-cloud-sql-connection-budget.mjs. The result stays within budget.

	Before	After
operatingMaximum	403	419
usableCeiling	490	490
remainingWithinUsableCeiling	87	71

What comes after merge

Nothing below runs from this PR. Each step is a separate production dispatch that needs its own explicit approval.

1. Re-read the director digest and the C27-C29 template digests. Stop and repin C30 if the director no longer serves the pinned digest.
2. Deploy Relay Asia Topology, mode plan, production, cell-ids=production-gce-c30, image set to the full C30 image in the file. This plan has not been run, because the live plan needs production credentials. Read its Plan: line and the step summary before any apply. It should show three additions for C30 and one change to the shared routing map, and it lists every cell whose committed image differed from its deployed one. If any other cell changes, the validator fails the plan, and that cell must be rolled first.
3. The same workflow, mode apply, with APPLY_RELAY_ASIA_TOPOLOGY.
4. Operate Relay Asia Admission, mode inspect, to read the live generation. Then mode register, cell-ids=production-gce-c30, with REGISTER_ASIA_MIGRATION_ONLY. C30 becomes migration-only.
5. The same workflow, mode configure, cell-ids=production-gce-c30, with CONFIGURE_ASIA_DIRECTOR, while regional rehome stays paused. This deploys a director whose cell list includes C30.
6. The same workflow, mode promote, cell-ids=production-gce-c30, with PROMOTE_ASIA_GENERAL and no evidence run. It promotes C30, runs the five-minute canary on it, and returns C30 to migration-only if the canary or its evidence fails.
7. After C30 serves, one follow-up adds C30 to the shadow gate's fleet pool list and moves it from the same-cap migration-only cells to the general ones.

Side effect on sealed evidence: this PR edits files that the evidence-provenance check pins. Those are the same-cap job, the same-cap wave list, and the rehome trust probe. Sealed same-cap canary authority or monitor evidence from before the merge will no longer match. Merge between same-cap batches, and plan a fresh monitor dry-run before resuming regional rehome.

Linked Issue

None.

Visual Proof

N/A. This PR has no UI. It changes Terraform inputs, operator scripts, and workflows.

Testing

• I manually tested these changes locally

• Automated tests added/updated, or explained why not below

• node --test dev/scripts/*.test.mjs from cloud/: 820 passed, 0 failed.

• pnpm --filter ./apps/relay-ops test: 12 files, 134 tests passed.

• pnpm typecheck and pnpm lint in cloud/: clean.

• terraform fmt -check -recursive in cloud/infra/terraform: clean.

• actionlint on the three edited workflows reports only the existing unknown runner-label warning.

• The root check:code-quality:changed gate skips cloud/ by design, so it checked nothing here.

• The overlay tests render the real startup-script template, so a template change that moves the image lines fails them.

• The workflow's state filter and the overlay command were run together on a synthetic state file that included a secret resource. The output held only template scripts, and only the drifted cell's image changed.

• The workflow reads the committed cell list from a quick plan that skips refresh and takes no state lock, not from terraform console. Without the outputs.tf change, console fails with Invalid index while C30 is declared but not yet created, because it evaluates every output against current state. The lead found that failure against production state. A throwaway Terraform 1.15.8 config showed the same console failure, while the targeted plan succeeded and gave the cell list exactly as written.

• New tests cover live cells pulled into the C30 plan, which pass unchanged and fail on update, delete, or replacement. They also cover the C30 canary evidence and its placement check, the canary and rollback steps running for C30, the rollback wave sets, the rule that C30 promotion needs all launch cells general, and the mixed Asia pool refusal.

• Not verified: a live C30 topology plan, which needs production credentials. The live digests above were read on 2026-09-22 and are not re-checked by this PR.

AI Disclosure

Written with Claude, an AI assistant, working from an operator's instructions. A separate AI review of the first revision found the plan scope and promotion gaps that the second commit fixes. A person reviews every production step before it is dispatched.

Review

The first revision was reviewed and returned as merge after fixes. This revision addresses these review findings:

• Plan scope. Live cells are planned at their deployed image, and any other change to them is rejected.
• Promotion proof. C30 runs the production canary with automatic rollback instead of relying on staging evidence.
• Shadow gate. C30 stays out of the fleet pool list until it serves.
• Budget. A test now covers the mixed Asia pool refusal.
• Rollback. Mixed partial rollback sets are rejected.

The second review also returned merge after fixes. This revision addresses:

• Console readers. The deployments output tolerates a declared but uncreated cell, so console calls in other workflows keep working between merge and C30's topology apply. A throwaway Terraform 1.15.8 config showed the warning on stdout without try and clean output with it. terraform validate passes.
• Same-cap admission. C30 is listed as migration-only until promotion. A test pins it as migration-only and refuses a wave that mixes it with C29.
• Operator notes. Merge between same-cap batches, because this PR changes files the evidence check pins. Watch C30's pool metrics for the first hour after promotion. New Asia hosts go to the least-loaded cell, so C30 takes most new placements until its load catches up.

C30 is now pinned to the director's live digest.

Please look hardest at two policies:

• Canary placement. The C30 canary depends on placement choosing the least-loaded general cell for a fresh host. If placement changes, the canary fails closed and C30 returns to migration-only.
• No input evidence. C30 promotion takes no input evidence. Its own canary runs right after the selector change and is the proof.

Agent skill upstream boundary

• Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Nothing here affects desktop, mobile, SSH, or other platforms. Old and new clients are unaffected because no wire format changes.

Checklist

• This PR is small and focused
• I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
• Before/after screenshots or videos attached for UI changes, or N/A with reason
• Self-reviewed for correctness, security, and performance
• Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
• pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

[pullfrog]

✅ No new issues found.

Reviewed changes — first review of PR #22375 at b5db1a64fe (5 commits, base ebed0964). Declares a disabled production-gce-c30 at the C27 shape and threads it additively through the Asia topology, admission, same-cap, rehome-trust, and canary-evidence tooling. I read the complete diff and traced the topology plan-scoping, admission promotion gating, and C30 canary binding paths end to end; the new tests pass (node --test dev/scripts/*.test.mjs → 799 pass; the single failure is a pre-existing missing @orca-cloud/relay-contract build, not this PR).

• Declared cell — production.tfvars adds production-gce-c30 (asia-east2-a, e2-standard-4, 30 GB, 6000 requests, pool 16, director-pinned image, disabled) and lists it as a rehome source; outputs.tf wraps the six per-cell lookups in try() so console and plans tolerate a declared-but-uncreated cell.
• Topology scoping — the new relay-live-cell-image-overlay.mjs plans each non-target cell at the image its state template actually serves, and validate-relay-asia-topology-plan.mjs rejects any mutating change to a cell outside the wave. The launch trio and C30 are separate reviewed waves; a declared-but-uncreated non-target cell fails the overlay closed.
• Admission waves — operate-relay-asia-admission.mjs accepts registration/promotion/rollback only as reviewed waves, requires C27–C29 general before promoting C30, and C30 takes no input evidence because its own canary is the proof with automatic rollback.
• Canary proof — load-relay-controls.mjs records the cells each control connected to, and relay-asia-rollout-evidence.mjs now binds the canary to its own cell's runtime metrics plus an exact placement-origin check.
• Same-cap / rehome / budget — C30 joins the migration-only same-cap list and rehome-trust allow-list, the Asia pool pin moves to per-environment (16 production / 10 staging), and the connection budget stays within the usable ceiling (419/490).

  ^{｜ View workflow run ｜ Using DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏}