Repository navigation
infra(relay): add staging cells c5-c7 at c2's shape for the dead-cell guard e2e - #27476
Conversation
…cell guard e2e Six feed cells, five in us-central1, so one dead US reserve cell is 1/6 of the fleet and 1/5 of its region and the dead-cell guards in #27458 can decide. Staging only; nothing is applied by merging.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Priority: ⬇️ Low Merge Risk: ⚪ Minimal · up to No actionable merge-blocking issue is identified. Review the staging plan before applying the configuration. Pre-merge checks |
|
ELI5
Staging gets three more small relay servers. The guard that decides when a relay server is really dead (#27458) never makes that call with fewer than 6 working servers to compare against, and staging has only 3. With these three added, the staging drill can show the guard actually moving desktops off a dead server.
What Changed
cloud/infra/terraform/environments/staging.tfvarsonly:staging-gce-c5(us-central1-a),staging-gce-c6(us-central1-b) andstaging-gce-c7(us-central1-f), each copying c2's shape:sha256:ac2ab357…), enabled at start (general).relay_region_rehome_source_cell_ids, as c2 is, so they serve the seat feed and the reserve/demote routes.Staging then has 6 cells with a seat feed (c1, c2, c4, c5, c6, c7), 5 of them in the US. One dead US cell is then 1/6 of the fleet and 1/5 of its region, so neither the fleet guard (more than ⅓) nor the region guard (½ or more) trips. c3 (old image, existing-only) still has no feed.
Merging applies nothing.
Why
#27458 refuses to call a cell dead unless at least 6 feed cells are polled. With 3, a staging test cannot tell a dead cell from a partition. A test-only override of the minimum would prove nothing about the real guards. With 3 cells it would also trip the region guard, since US would have two cells and one dead is half.
I copied c2's shape, not the reviewed Asia-topology shape (e2-standard-4, cap 3000, migration-only at start). These cells exist to be ordinary US general cells for the drill.
How to apply (staging grant; read the plan first)
The reviewed
cloud-deploy-relay-asia-topology.ymlpath cannot add these cells. Its scripts accept only cap 3000 / e2-standard-4 / migration-only cells, and staging admission is wired to a C4-only proof route. So the driver applies with targeted Terraform:Plan with these targets, for each of
staging-gce-c5,staging-gce-c6andstaging-gce-c7:google_compute_instance_template.relay_gce_cell["…"]google_compute_instance_group_manager.relay_gce_cell["…"]google_compute_backend_service.relay_gce_cell["…"]Also target
google_compute_url_map.relay_gce[0]. The wildcard DNS record and wildcard certificate already coverc5–c7.relay-staging.onorca.dev.Check that the plan touches no existing cell's template or MIG. Other cells must plan at the image they serve (see the live-image overlay in the topology workflow).
Once the cells are healthy, add them to the director.
ORCA_RELAY_CELLS_JSONis ingoogle_cloud_run_v2_service.relay's Terraform template, and the blue/green deploy script refuses a cell-set change. So this is a targeted plan of the director service, which rolls a new director revision.ORCA_RELAY_CELLS_JSON. In particular, watch scaling: staging.tfvars says min 0, while the reserve runs use 2. Also watch traffic and the reserve-placement and seat-feed settings (those are read from the serving revision).After the director restart, confirm three things:
generalinrelay_cell_admission;inspectstill verifies;orca_relay_shadow_seat_summaryshowslive ≥ 6.Linked Issue
N/A (internal staging test topology).
Visual Proof
N/A: Terraform variables for staging.
Testing
terraform fmt -checkandterraform validatepass.Review
Agent skill upstream boundary
Notes
Staging only. Production tfvars are untouched. Cost: three e2-standard-2 VMs (6 vCPU), within staging quota (6 of 3,000 CPUs in use in us-central1).
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)