[Jinwoo-H]

	Files	Added	Deleted	Net
Test	0	0	0	0
Prod	9	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​827	0	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​827

ELI5

Some Asia desktops lose their relay connection a few times a day, all in the same second. Google's load balancer, which sits in front of every relay cell, ends those connections; it happens only for the Asia cells. This PR builds a second front door for one Asia cell, c35, that we run ourselves, with no Google proxy in it. Nothing uses the new door until we change one DNS record, and deleting that record is the undo. Then we can see whether c35 stops dropping when the other Asia cells drop.

What Changed

• Before:
  • c35.relay.onorca.dev resolves, through the *.relay.onorca.dev wildcard, to Google's global load balancer.
  • Google terminates TLS there and proxies to the c35 VM.
  • When the asia-east2 proxy tier fails, every Asia cell loses 8–18% of its streams in one second (statusDetails=internal_error).
• After this PR is applied, with DNS unchanged: nothing changes for any user. A new VM, orca-cloud-relay-gce-edge-c35, stands next to c35 with a static IP, and serves c35's hostname with its own certificate.
• After the separate DNS step (an exact c35 A record in Cloudflare): c35's desktops connect to that VM. Envoy terminates TLS there and forwards over the VPC to the same c35 VM. The cell stays behind the Google load balancer the whole time, so both paths serve during and after the switch.
• The mechanism (relay-gce-passthrough-edge.tf, keyed by relay_gce_passthrough_edge_cells). Three containers run on a COS VM:
  • Envoy (TLS and WebSocket). It mirrors the load balancer: the same 86,400 s stream lifetime, no idle timeout, 404 for any other host, and 503 when the cell has no VM. It rewrites X-Forwarded-For to client, edge-IP, the shape the cell's per-source admission reads.
  • lego (Let's Encrypt, DNS-01). It answers into a Cloud DNS zone that only this edge may edit, and that Cloudflare delegates. Renewals swap in without dropping streams.
  • A discovery loop. It follows the cell VM by label, because GCE refuses an internal load balancer on an instance group the global LB already balances.
• Monitoring: a renewal-failure alert, and an uptime check that alerts at < 21 days to certificate expiry.
• Config: production lists c35; staging lists c4 (same zone, asia-east2-a), which carries the proof.
• Runbook: cloud/docs/relay-passthrough-edge.md.

Why

• Why an edge VM and not TLS in the relay or a sidecar on the cell. Both of those change the cell's instance template. A template change has no fast production path: the same-cap validator requires the startup script to stay byte-equal apart from pinned fields. It would also make c35 permanently different on every future roll.
• What the edge leaves alone. The cell, its image, its MIG and every rollout, drain, flag and monitor workflow stay untouched. The edge is inert until DNS moves.
• Why DNS-01 into a delegated zone. Google-managed certificates cannot be exported. HTTP or TLS challenges would leave a TLS gap at cutover. A Cloudflare token on a VM could rewrite every onorca.dev record, including login.
• Why not a regional Google load balancer. It is still a Google proxy, so it would not test the hypothesis.
• Targeted plans stay off the cell. The edge references the cell's instance group by name, so a targeted edge plan never pulls in the cell's MIG or template. tfvars images are stale by design, so pulling them in would recreate c35.

Linked Issue

N/A (maintainer; internal Asia relay drop investigation).

Visual Proof

N/A. This is infrastructure with no UI; the evidence is under Testing.

Testing

• Read-only production plan (-lock=false, targeted as in the runbook): 15 to add, 0 to change, 0 to destroy, with no cell, URL map or address resource in it.

• Staging, edge in front of c4. Applied with targeted staging applies only; staging-only changes.

  • /health through the edge returns 200, from the cell. The LB path stays 200 at the same time.
  • Desktop-equivalent WebSocket client (Node ws, SNI c4, pinned to the edge IP):
    • /v1/connect/<id> upgrades with 101;
    • the relay answers relay-hello and closes 4401, identical to the LB path;
    • /v1/host/control without auth returns 401 on both paths;
    • a throwaway client CA was used until the public certificate exists.
  • Headers: a caller-supplied X-Forwarded-For and x-envoy-* header were dropped. The upstream saw exactly client, edge-IP. The cell's per-source limiter returned 429 to a burst from one client through the edge.
  • Unknown host through the edge returns 404.
  • Certificate swap under load. Streams opened before the rename closed normally (relay 4401 at 2.8 s, not 1006), and the next connection presented the new certificate.
  • c4 MIG recreate. The edge stayed pointed at the cell and served again as soon as the new VM was up.
  • Edge VM reset. It served again 19 s after the reset, with the certificate kept on disk.
  • Public certificate and DNS cutover/rollback: pending two staging Cloudflare records (NS for relay-staging-acme.onorca.dev, CNAME for _acme-challenge.c4.relay-staging.onorca.dev). The renewer currently logs that the CNAME is missing and places no order.

• Script tests. node --test dev/scripts/*.test.mjs: 1014 pass, 4 fail. The failures do not involve this change: the relay contract package is unbuilt, two suites need the tsx runner, and an argv test fails on a workflow this PR does not touch. The root-partition fixture includes the new families. terraform validate passes.

• I manually tested these changes locally

• Automated tests added/updated, or explained why not below

The behaviour is Terraform plus shell. It is covered by the staging proof above and the root-partition test, not by unit tests.

AI Disclosure

Review

Do not apply to production from this PR. The production steps are a targeted apply, then two inert Cloudflare records, then a probe, then the A record. They are in the runbook, and each needs Jinwoo's go.

Agent skill upstream boundary

• Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

• Security:
  • The edge's service account has log write, read-only compute, and DNS admin on its own ACME zone only.
  • No client address or host ID appears in the edge logs.
  • Envoy's admin port sits on a private Docker network.
  • The images are pinned by digest from Docker Hub; mirroring them to Artifact Registry is a follow-up before any wider rollout.
• Compatibility: no client, director or wire change. The hostname and SNI are the same, and Let's Encrypt is trusted by desktop and mobile.
• SSH and remote: unaffected.

Checklist

• This PR is small and focused
• I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
• Before/after screenshots or videos attached for UI changes, or N/A with reason
• Self-reviewed for correctness, security, and performance
• Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
• pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 214eaaed-d51c-4617-ada3-24fdbb4c754c

📥 Commits

Reviewing files that changed from the base of the PR and between 9be9d11 and 3921972.

📒 Files selected for processing: 9

• cloud/dev/fixtures/terraform-root-partition/families.json
• cloud/docs/relay-passthrough-edge.md
• cloud/infra/terraform/environments/production.tfvars
• cloud/infra/terraform/environments/staging.tfvars
• cloud/infra/terraform/outputs.tf
• cloud/infra/terraform/relay-gce-edge-envoy.yaml.tftpl
• cloud/infra/terraform/relay-gce-edge-startup.sh.tftpl
• cloud/infra/terraform/relay-gce-passthrough-edge.tf
• cloud/infra/terraform/variables.tf

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 6 remain after this review.

📝 Walkthrough

The changes add optional Relay GCE passthrough edges for configured cells. Terraform provisions edge VMs, network access, ACME DNS resources, outputs, and monitoring. Startup scripts run Envoy, renew certificates, and discover cell endpoints. Staging and production configuration select edge cells and ACME domains. Documentation describes deployment, validation, rollback, monitoring, and maintenance, and states that staging did not verify public-certificate issuance or DNS cutover.

Priority: ➖ Normal

Merge Risk: ⚪ Minimal · up to 39219

No confirmed defect remains to block merge. Public-certificate issuance and DNS cutover still need the staged validation described in the runbook before activation.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly identifies the main change: an Orca-run TLS edge for one relay cell that remains inactive until DNS changes.
Description check		The description follows the required template and provides clear ELI5, change, rationale, testing, security, compatibility, review, and checklist details. It also explains the maintainer exception for…
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[OrcaWin]

Closing because this public-repository Terraform experiment is superseded by the deployment ownership move in #31792 (307f0cdaeed), and the proposal itself records the experiment as shelved.

I read the complete nine-file patch, body, runbook, and discussion. All nine target paths—the production/staging Terraform root, edge VM/firewall/DNS/monitoring resources, Envoy and startup templates, fixture inventory, and runbook—are absent from current main (ad2156c3584). cloud/README.md now assigns production/staging Terraform, operator tools, runbooks, and deployment workflows to private stablyai/orca-cloud; the public runtime remains separately maintained.

The proposed runbook explicitly says “Status: shelved (2026-10-16). Nothing here is applied,” notes that staging resources were destroyed, and distinguishes its throwaway-CA proof from unproven public-certificate/DNS cutover. This closure does not assert that c35 was switched, that the Asia proxy-drop problem is solved, or that any corresponding private deployment change landed. Any revival belongs in the current private deployment owner rather than resurrecting the removed public Terraform root.