Skip to content

infra(relay): add staging cells c5-c7 at c2's shape for the dead-cell guard e2e - #27476

Merged
Jinwoo-H merged 1 commit into
mainfrom
infra-staging-relay-cells-c5-c7
Oct 11, 2026
Merged

Jinwoo-H merged 1 commit into
mainfrom
infra-staging-relay-cells-c5-c7

Conversation

@Jinwoo-H

@Jinwoo-H Jinwoo-H commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

ELI5

Staging gets three more small relay servers. The guard that decides when a relay server is really dead (#27458) never makes that call with fewer than 6 working servers to compare against, and staging has only 3. With these three added, the staging drill can show the guard actually moving desktops off a dead server.

What Changed

cloud/infra/terraform/environments/staging.tfvars only:

  • New cells: staging-gce-c5 (us-central1-a), staging-gce-c6 (us-central1-b) and staging-gce-c7 (us-central1-f), each copying c2's shape:
    • e2-standard-2, 30 GB disk, COS 121-18867-528-7;
    • capacity 4000, hard cap 600, unobserved bound 60;
    • image D (sha256:ac2ab357…), enabled at start (general).
  • Rehome trust: all three are added to relay_region_rehome_source_cell_ids, as c2 is, so they serve the seat feed and the reserve/demote routes.

Staging then has 6 cells with a seat feed (c1, c2, c4, c5, c6, c7), 5 of them in the US. One dead US cell is then 1/6 of the fleet and 1/5 of its region, so neither the fleet guard (more than ⅓) nor the region guard (½ or more) trips. c3 (old image, existing-only) still has no feed.

Merging applies nothing.

Why

#27458 refuses to call a cell dead unless at least 6 feed cells are polled. With 3, a staging test cannot tell a dead cell from a partition. A test-only override of the minimum would prove nothing about the real guards. With 3 cells it would also trip the region guard, since US would have two cells and one dead is half.

I copied c2's shape, not the reviewed Asia-topology shape (e2-standard-4, cap 3000, migration-only at start). These cells exist to be ordinary US general cells for the drill.

How to apply (staging grant; read the plan first)

The reviewed cloud-deploy-relay-asia-topology.yml path cannot add these cells. Its scripts accept only cap 3000 / e2-standard-4 / migration-only cells, and staging admission is wired to a C4-only proof route. So the driver applies with targeted Terraform:

  1. Plan with these targets, for each of staging-gce-c5, staging-gce-c6 and staging-gce-c7:

    • google_compute_instance_template.relay_gce_cell["…"]
    • google_compute_instance_group_manager.relay_gce_cell["…"]
    • google_compute_backend_service.relay_gce_cell["…"]

    Also target google_compute_url_map.relay_gce[0]. The wildcard DNS record and wildcard certificate already cover c5–c7.relay-staging.onorca.dev.

  2. Check that the plan touches no existing cell's template or MIG. Other cells must plan at the image they serve (see the live-image overlay in the topology workflow).

  3. Once the cells are healthy, add them to the director. ORCA_RELAY_CELLS_JSON is in google_cloud_run_v2_service.relay's Terraform template, and the blue/green deploy script refuses a cell-set change. So this is a targeted plan of the director service, which rolls a new director revision.

    • Stop if that plan changes anything besides ORCA_RELAY_CELLS_JSON. In particular, watch scaling: staging.tfvars says min 0, while the reserve runs use 2. Also watch traffic and the reserve-placement and seat-feed settings (those are read from the serving revision).
  4. After the director restart, confirm three things:

    • the new cells read general in relay_cell_admission;
    • the admission selector inspect still verifies;
    • the director's orca_relay_shadow_seat_summary shows live ≥ 6.

Linked Issue

N/A (internal staging test topology).

Visual Proof

N/A: Terraform variables for staging.

Testing

  • terraform fmt -check and terraform validate pass.
  • The staging plan is read by the driver before apply (above).

Review

Agent skill upstream boundary

  • Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Staging only. Production tfvars are untouched. Cost: three e2-standard-2 VMs (6 vCPU), within staging quota (6 of 3,000 CPUs in use in us-central1).

Checklist

  • This PR is small and focused
  • I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
  • Before/after screenshots or videos attached for UI changes, or N/A with reason
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
  • pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

…cell guard e2e

Six feed cells, five in us-central1, so one dead US reserve cell is 1/6 of the fleet and 1/5 of its
region and the dead-cell guards in #27458 can decide. Staging only; nothing is applied by merging.
@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 59257f7d-ec90-48f9-b834-a22ac90e197f


📥 Commits

Reviewing files that changed from the base of the PR and between d3417b2 and e678193.



📒 Files selected for processing (1)
  • cloud/infra/terraform/environments/staging.tfvars


Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.




📝 Walkthrough
📝 Walkthrough

Walkthrough

Staging configuration adds GCE cells c5–c7, each with a 4,000-request capacity, a 600-connection hard cap, and a 60-connection unobserved bound. The rehome-trust source list expands to include cells c1–c7.



Priority: ⬇️ Low

Merge Risk: ⚪ Minimal · up to e6781

No actionable merge-blocking issue is identified. Review the staging plan before applying the configuration.

Pre-merge checks | Passed 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly identifies the staging infrastructure change, the cells added, their configuration basis, and the dead-cell guard end-to-end test purpose.
Description check Passed The description is detailed and covers the change, rationale, deployment procedure, testing status, scope, cost, and checklist. The Linked Issue section says N/A even though the description references…

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Jinwoo-H
Jinwoo-H merged commit 741f953 into main Oct 11, 2026
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant