Skip to content

fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), adoptable deploy shape, event-loop stall alert - #27406

Merged
Jinwoo-H merged 3 commits into
mainfrom
relay-director-cloud-sql-proxy-sidecar
Oct 11, 2026
Merged

Jinwoo-H merged 3 commits into
mainfrom
relay-director-cloud-sql-proxy-sidecar

Conversation

@Jinwoo-H

@Jinwoo-H Jinwoo-H commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​114 $\color{#cf222e}{\Huge{\mathbf{−}}}$​5 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​109
Prod 9 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​324 $\color{#cf222e}{\Huge{\mathbf{−}}}$​14 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​310

ELI5

While its database is starting up or failing over, the relay director freezes. Every attempt to reach the database stops the whole process for up to 10 seconds, and the attempts follow each other back to back. This PR gives the director the same small database proxy the relay cells already run beside them. With it, a database outage makes the director answer "try again" within a couple of seconds instead of going silent.

It's switched on in staging only. Production plans don't change until we flip it there, and the rollout steps for both are written out below.

What Changed

Two new Terraform variables, both off by default. staging.tfvars turns both on.

  • relay_director_cloud_sql_proxy_sidecar. The director's Cloud Run template gains a cloud-sql-proxy sidecar.
    • It uses the cells' digest-pinned image, relay_gce_cloud_sql_proxy_image (v2.18.2), and listens on 127.0.0.1:5432. The cells run the same proxy beside the relay, but over a Unix socket on a full Linux kernel.
    • /startup startup probe, and a /liveness liveness probe every 10 s with 3 failures allowed.
      • Per the proxy's --help, /startup doesn't wait for the database, so the director still boots during an outage.
      • /liveness fails only when the proxy itself is wedged.
    • The relay container is named relay, stays containers[0], and depends_on the proxy.
    • The /cloudsql volume and its mount are dropped.
    • DATABASE_URL reads a new secret, orca-cloud-relay-director-database-url (postgresql://…@127.0.0.1:5432/orca_relay), readable by the director runtime account. Cells keep the socket URL secret.
    • The proxy authenticates as the revision's service account, which already has roles/cloudsql.client (relay.tf:23). It dials the public IP, as today's built-in connector does.
    • It runs with GOMAXPROCS=1. Go would otherwise size its scheduler to the host's cores rather than the 0.25 vCPU quota, and get throttled in bursts.
    • Cost. The sidecar gets 0.25 vCPU and 256 MiB. Cloud Run rejects only a total under 1 vCPU with concurrency above 1. I checked that against the Cloud Run API with validateOnly: relay 1 vCPU plus proxy 0.25/0.5/0.08 with concurrency 1000 is accepted, and a relay at 0.5 vCPU is rejected with "Total cpu < 1 is not supported with concurrency > 1". So director compute goes up 25%, not 100%.
  • relay_event_loop_stall_alert. Adds a counter metric, orca_relay_event_loop_stalls, that counts runtime samples with jsonPayload.eventLoopDelayMsMax>=5000, plus an Orca Relay: event loop stall alert on any count. It follows the relay_long_cell_inventory_hold pattern: the filter does the threshold, so any value is a breach. There is no percentile over a max. The alert doesn't page yet. The field comes from fix(relay): report event-loop freezes and keep seat polls off the cell-list read #27405.

A reviewed one-time shape transition in the deploy script.

  • Problem. The director service declares no traffic block, so an apply creates a new revision at 0%. The next deploy then failed its shape guard. assertPreservedRevisionShape compares every candidate with the serving revision, and the sidecar, the dropped volume, the secret reference and the container name all differ.
  • Fix. deploy-relay-blue-green.mjs --shape-baseline-revision <name> makes the guard compare against the named revision instead.
    • It must be the service's latest created revision.
    • It must not be serving.
    • It must run the serving image. Terraform ignores the image, so anything else wasn't an apply.
    • The deploy itself must also keep the serving image (--image digest equal to the serving digest). A shape change and an image change never ride together.
    • parseArguments refuses to combine it with --bootstrap-runtime-identity true, a topology change (--capacity-cell-id or --director-cells-json) or --prune-revisions true. That covers every caller, not just the workflows.
    • Everything else the guard checks is unchanged.
    • Not checked: who created the revision. The serving.knative.dev/creator annotation names the principal, not the tool. In staging, deploy-script revisions and workflow Terraform applies both run as orca-cloud-staging-gha-relay, and operator gcloud revisions show jinwoo@stably.ai with client-name: gcloud. A creator check would either block a legitimate workflow apply or pass a hand-made revision. The latest-created, unserved and same-image checks bound the choice instead.
  • Workflows. Both the staging and production director workflows take a shape-baseline-revision input. It is validated against the service's revision-name pattern, needs the confirmation ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE, and refuses to combine with bootstrap, prune or a regional-placement change. When the input is empty, both workflows behave exactly as before.

Staging drift pinned.

  • What drifted. The serving staging director runs concurrency 1000 and a 3600 s timeout. Terraform's defaults are 80 and 30 s, and main's staging plan already carried that change.
  • Why it matters now. The deploy now adopts the apply's shape, so applying would lower both.
  • Fix. staging.tfvars pins both to what is serving, so the apply leaves them as they are.

The five new resource families are registered in the Terraform root-partition fixture.

Why

Root cause. Cloud Run's first-generation built-in Cloud SQL connection embeds the Auth Proxy behind the /cloudsql socket. On that socket, connect() itself blocks until the proxy's dial to the instance ends, and while an instance is starting that dial times out after 10 s on each try. The director's awaits run back to back without returning to the event loop, so no timer fires. That covers the pool's 2 s connect timeout, the 4 s assign queue wait, metrics, and the seat polls that keep reserve cells fed.

What it cost.

  • Staging, 2026-10-11 06:12:14–06:14:28 UTC. Both directors froze for about 134 s, assigns took up to 165 s, and the reserve cell's dead-man tripped.
  • Production, 2026-09-28 about 14:09–14:11 UTC (the only Cloud SQL connector failures in 14 days):
    • Four of five directors froze for 40–51 s each.
    • Their event-loop p99 read 3–7 s, and their connect attempts took 10–20 s against a 2 s timeout.
    • The one director with no connector failures stayed healthy.

Why the sidecar. It accepts a loopback TCP connection straight away and dials in the background. The pool's 2 s timeout then works, and a failed attempt costs the loop nothing.

Alternatives considered.

  • Second-generation execution environment. It still uses Cloud Run's managed connection, and nothing shows its socket behaves differently.
  • In-process @google-cloud/cloud-sql-connector. A new dependency, with TLS handled in the relay.
  • A circuit breaker. Every probe would still freeze the loop for 10 s.
  • A sidecar-only shape normaliser in the deploy script (what the reviewer suggested). It would not cover the other differences a director apply carries. Staging adds PUBLIC_STICKY_*, PUBLIC_ASSIGNMENTS_ENABLED, and empty FENCE/MONITOR service-account variables; production adds an empty ASIA_PROOF_SERVICE_ACCOUNT. Each of those also fails the guard. Naming the reviewed apply revision covers the whole reviewed plan, and nothing outside it.

Staging enablement sequence (after merge, with Jinwoo's go)

  1. Wake staging if it's asleep (Power Relay Staging, wake, wake-cells: all). Confirm Cloud SQL is RUNNABLE.

  2. Record the pre-state:

    • the serving revision (gcloud run services describe orca-cloud-relay-staging --project onorca-cloud-staging --region us-central1)
    • its image digest, min/max scale, ORCA_RELAY_CELLS_JSON, ORCA_RELAY_RESERVE_PLACEMENT and ORCA_RELAY_SHADOW_SEAT_FEED_CELLS
    • the current rehome generation
  3. Plan and review. From cloud/infra/terraform, with a GOOGLE_OAUTH_ACCESS_TOKEN from gcloud auth print-access-token, run a saved targeted plan:

    terraform init -reconfigure -backend-config=backend/staging.hcl
    terraform plan -var-file=environments/staging.tfvars \
      -target=google_cloud_run_v2_service.relay \
      -target=google_secret_manager_secret.relay_director_database_url \
      -target=google_secret_manager_secret_version.relay_director_database_url \
      -target=google_secret_manager_secret_iam_member.relay_director_database_url_accessor \
      -target=google_logging_metric.relay_event_loop_stall \
      -target=google_monitoring_alert_policy.relay_event_loop_stall \
      -out=director-sidecar.tfplan
    

    Expect Plan: 5 to add, 1 to change, 0 to destroy. The read-only plan from this branch gives exactly that.

    • The 5 creates: the secret, its version, its accessor, the metric and the alert.
    • The director update. Only these changes should appear:
      • the sidecar, plus the relay name and depends_on
      • the DATABASE_URL secret swap, and the /cloudsql volume and mount dropped
      • the drift already on main, which the deploy restores or which is behaviour-neutral:
        • ORCA_RELAY_IMAGE_DIGEST removed
        • ORCA_RELAY_CELLS_JSON re-ordered with the same value
        • PUBLIC_STICKY_* and PUBLIC_ASSIGNMENTS_ENABLED at their app defaults
        • empty FENCE/MONITOR service accounts
        • template min instances 2→0 and the revision field
    • Stop if concurrency, timeout, image or anything else shows.
  4. Apply the saved plan with the state lock: terraform apply director-sidecar.tfplan.

  5. Record the 0% revision. Corrected after the staging run (2026-10-11, see docs(relay): director Cloud SQL sidecar enablement and rollback #27461):

    • status.latestCreatedRevisionName is a new revision, N, and traffic is still 100% on the step-2 revision.
    • Cloud Run does not start N: it reads Ready/Retired and logs nothing. The first real sidecar boot is the deploy's selector-rollback candidate in step 6, health-checked before traffic moves.
    • The first apply may also fail on the alert policy with 404 Cannot find metric(s): the new metric takes minutes to appear. If so, re-plan and apply the policy alone a few minutes later.
  6. Dispatch Deploy Relay Staging with:

    • expected-image-digest = the step-2 digest
    • reserve-placement: preserve
    • expected-rehome-generation = the step-2 value
    • bootstrap-runtime-identity: false
    • capacity-cell-id: none
    • min-instances: preserve
    • shadow-seat-feed-cells: preserve
    • shape-baseline-revision: N
    • confirmation: ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE
  7. Verify the new serving revision:

    • two containers, with relay first
    • DATABASE_URL set to orca-cloud-relay-director-database-url:latest
    • ORCA_RELAY_IMAGE_DIGEST back, minScale 2, and the step-2 reserve and shadow settings unchanged
    • no cloudsql-instances annotation
    • logs: the proxy reports listening on 127.0.0.1:5432, the director reaches /ready, there are no new orca_relay_postgres_query_failed lines, and seat summaries and placement summaries keep flowing
  8. Run the Cloud SQL stop→start drill. Pass means:

  9. Drill measurement, sidecar CPU. Before step 4, run the same assign burst (the staging load workflow's assign phase, same profile both times) on the socket revision, and again after step 7 on the sidecar revision. Record from runtime_metrics:

    • sql_latency_ms p95 and max
    • the director's databasePoolWaitMsMax

    Also record the cloud-sql-proxy container's CPU from run.googleapis.com/container/cpu/usage with metric.label.container_name="cloud-sql-proxy", as a fraction of its 0.25 vCPU limit. That is the per-container metric; container/cpu/utilizations carries no container label. Rule: if proxy utilization nears its 0.25 limit (p99 above about 80%), or SQL latency p95 rises past the socket run's, raise the sidecar to 0.5 vCPU in a follow-up before production.

  10. Rollback at any point after step 6. Re-dispatch the deploy with the step-2 digest after a reviewed apply with the variable set to false, again using shape-baseline-revision for the apply's revision. The step-2 revision and the socket secret are untouched throughout.

Emergency socket rollback. If the sidecar revision misbehaves and the reviewed rollback above is too slow, move traffic straight back to the pre-sidecar revision. That revision survives the deploy because the shape-adoption deploy refuses --prune-revisions, and it still carries its own /cloudsql connection and socket URL secret:

gcloud run services update-traffic <service> --to-revisions=<pre-sidecar revision>=100

No repository workflow owns an arbitrary-revision traffic move today. The deploy script only moves traffic to its own tagged candidate or rollback. So this is an operator action with run.admin: in production it needs Jinwoo's go and the production rollout lease. Afterwards the service template is still the sidecar shape. The next deploy refuses (serving revision on the socket shape, template on the sidecar shape) until a reviewed apply with the variable set to false and a deploy that adopts that apply's revision.

Production enablement path (not run; for review)

  1. Prove it in staging first. Open a reviewed PR that sets relay_director_cloud_sql_proxy_sidecar = true (and the alert variable) in production.tfvars, after the staging drill passes.
  2. Expect this plan. The production plan from this branch with both set gives exactly Plan: 5 to add, 1 to change, 0 to destroy, with the same targets as staging step 3.
    • The director update carries the drift already on main:
      • ORCA_RELAY_IMAGE_DIGEST removed (the deploy restores it)
      • ORCA_RELAY_CELLS_JSON re-ordered with the same value
      • an empty ORCA_RELAY_ASIA_PROOF_SERVICE_ACCOUNT added
      • the template revision and service-level scaling fields cleared
    • The sidecar changes are the same as staging's.
    • Not in production's plan: concurrency and timeout. Production already serves the Terraform values.
  3. Apply under the production Cloud SQL rollout lease. Same checks as staging step 5. Production min instances are 5. The apply's revision is at 0%, so the serving revision keeps its floor.
  4. Dispatch Deploy Relay Production Director. Use the serving digest, bootstrap-runtime-identity: false, prune-incompatible-revisions: false, regional-placement-mode: preserve, shape-baseline-revision: <apply revision> and confirmation: ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE. The deploy is blue/green and keeps the serving floor, so nothing disconnects.
  5. Verify as in staging step 7. Watch Orca Relay: event loop stall and the 30 s flush cadence across the next Cloud SQL maintenance window.

Linked Issue

N/A (internal relay incident follow-up).

Visual Proof

N/A: infrastructure only.

Testing

  • terraform fmt -check -recursive and terraform validate pass.
  • Production, current flags (read-only -lock=false plans, refreshed). Targeted at every resource this PR touches, branch and main give an identical plan: Plan: 10 to add, 2 to change, 0 to destroy, with the same action list and the same rendered diff. This PR adds 0 production changes.
    • The 10/2 is existing drift: ten runtime metrics not yet applied, the connection_headroom alert, and the director's deploy-owned fields.
    • The narrow enablement targets give 0 to add, 1 to change, and that one change is the same existing director drift.
  • Production with both variables set (simulated with -var, narrow targets): 5 to add, 1 to change, as listed under the production path.
  • Staging (narrow targets): 5 to add, 1 to change. The director update no longer touches concurrency or timeout.
  • Cloud Run validateOnly on staging (nothing persisted):
    • Accepted: the relay container with a sidecar at 0.25, 0.5 or 0.08 vCPU, concurrency 1000, a /liveness probe and depends_on.
    • Rejected, as expected: a relay at 0.5 vCPU with concurrency 1000, and a probe with two probe types. These prove the API really validates.
  • deploy-relay-blue-green.test.mjs: 50 pass. New cases:
    • After a sidecar apply, a deploy without the flag refuses the shape and traffic doesn't move.
    • With the apply revision named, the rollback and the candidate both adopt the sidecar shape.
    • The flag is refused for a stale revision, for the serving revision and for a different image.
    • Argument validation.
  • Other dev-script tests: the workflow contract, partition and alert suites pass, 959 of 960 dev-script tests in all. The one failure, relay-load-control-peer, needs an unbuilt package in this lane and passes where the packages are built.
  • actionlint on both changed workflows reports nothing new; it only flags the existing blacksmith runner label.

AI Disclosure

N/A (internal contributor).

Agent skill upstream boundary

  • Not applicable.

Notes

  • Needs Jinwoo's go to merge (shared Terraform root and both director workflows).
  • The /cloudsql annotation and dependency annotation names come from the live revision's v1 shape. The deploy no longer compares them across the transition, so a naming surprise can't wedge it; step 7 confirms what Cloud Run actually wrote.

Checklist

  • This PR is small and focused
  • I explained what changed and why
  • N/A: no UI
  • Self-reviewed for correctness, security, and performance
  • Cross-platform, SSH/remote impact: N/A (relay infrastructure only)
  • Terraform validate, fmt and the cloud dev-script tests pass locally

…ng), gated event-loop stall alert

On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole
event loop until the connector's upstream dial ends, 10 s per attempt while the
instance starts. Staging froze both directors for 134 s on 2026-10-11 and
production four directors for 40-50 s on 2026-09-28.

relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the
director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP
database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max
metric and a 5 s stall alert. Both default off, so production plans unchanged;
staging turns both on.
@coderabbitai

coderabbitai Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

📝 Walkthrough
📝 Walkthrough
📝 Walkthrough

Walkthrough

Terraform adds an optional Cloud SQL Auth Proxy sidecar for the relay director, with a dedicated database URL secret and conditional service wiring. It also adds an optional event-loop stall metric and alert. Staging enables both options. The deployment workflows and script add an optional shape-baseline revision and use it to validate and compare director revision transport shapes.





Priority: ➖ Normal

Merge Risk: 🟡 Moderate · up to d7b8b

The new event-loop stall alert would not fire because the relay runtime does not emit the field the metric filters on. Emit eventLoopDelayMsMax from the runtime before relying on this alert. The sidecar and deploy changes are otherwise bounded.

Pre-merge checks | Passed 4 | Failed 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. (4 skipped: 4… Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly summarizes the primary changes: the director Cloud SQL proxy sidecar, adoptable deployment shape, and event-loop stall alert.
Description check Passed The description is complete and structured according to the template. It explains the problem, changes, rationale, rollout plans, testing, limitations, and required approvals. It correctly marks visua…

Full details: Docstring Coverage

Explanation

Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 2 files. (4 skipped: 4 unsupported.)


  • Fix all pre-merge checks with AI
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Commit to this branch
  • Create a new PR








  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 28079f63-3e5d-438a-9856-0e07ec443bf7
📥 Commits

Reviewing files that changed from the base of the PR and between b197bc6 and 205844f.

📒 Files selected for processing (6)
  • cloud/dev/fixtures/terraform-root-partition/families.json
  • cloud/infra/terraform/environments/staging.tfvars
  • cloud/infra/terraform/relay-database.tf
  • cloud/infra/terraform/relay-observability.tf
  • cloud/infra/terraform/relay.tf
  • cloud/infra/terraform/variables.tf

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.

threshold_run = 5000
threshold_gce = 5000
duration = "0s"
aligner = "ALIGN_PERCENTILE_99"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

🔎 Supported by static analysis

🏁 Script executed:

set -eu
printf '%s\n' '--- diff ---'
git diff --no-ext-diff --unified=80 08c8d30f105f96093ca547d36764e4aa266b92cf 205844f600b8a25c5b0283533da2bc4e4ba716af -- cloud/infra/terraform/relay-observability.tf cloud/infra/terraform | sed -n '1,360p'
printf '%s\n' '--- alert/metric references ---'
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' . || test "$?" -eq 1
printf '%s\n' '--- relevant resource blocks ---'
nl -ba cloud/infra/terraform/relay-observability.tf | sed -n '110,215p'
printf '%s\n' '--- metric-name references ---'
rg -n -F --glob '!**/.git/**' -- 'event_loop_ms_max' . || test "$?" -eq 1

Repository: stablyai/orca

Length of output: 37228


🏁 Script executed:

set -eu
git diff --no-ext-diff --unified=80 08c8d30f105f96093ca547d36764e4aa266b92cf 205844f600b8a25c5b0283533da2bc4e4ba716af -- cloud/infra/terraform/relay-observability.tf
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' . || test "$?" -eq 1
nl -ba cloud/infra/terraform/relay-observability.tf | sed -n '110,215p'
rg -n -F --glob '!**/.git/**' -- 'event_loop_ms_max' . || test "$?" -eq 1

Repository: stablyai/orca

Length of output: 40632


🏁 Script executed:

set -eu
printf '%s\n' '--- runtime metric wiring ---'
rg -n -F --glob '!**/.git/**' -- 'relay_runtime_log_filter' cloud/infra/terraform
rg -n -F --glob '!**/.git/**' -- '30-second runtime sample' cloud/infra/terraform
rg -n -F --glob '!**/.git/**' -- 'relay_snapshot' cloud/infra/terraform
printf '%s\n' '--- event-loop field variants ---'
rg -n -i --glob '!**/.git/**' -- 'event.?loop|eventLoopDelay|runtime snapshot|runtime metrics' src cloud dev tests 2>/dev/null || test "$?" -eq 1
printf '%s\n' '--- relay rollout limits ---'
rg -n -F --glob '*.tfvars' -- 'relay_max_instances' cloud/infra/terraform
rg -n -F --glob '*.tfvars' -- 'relay_min_instances' cloud/infra/terraform

Repository: stablyai/orca

Length of output: 41293


🏁 Script executed:

set -eu
printf '%s\n' '--- relay runtime emitter ---'
nl -ba cloud/apps/relay/src/relay-observability.ts | sed -n '430,615p'
printf '%s\n' '--- flag contract ---'
nl -ba cloud/infra/terraform/variables.tf | sed -n '475,500p'
printf '%s\n' '--- cadence reference ---'
nl -ba cloud/dev/scripts/relay-same-cap-shadow-gate-verdict.mjs | sed -n '135,165p'
printf '%s\n' '--- exact max-field references ---'
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' cloud/apps cloud/dev cloud/infra tests || test "$?" -eq 1

Repository: stablyai/orca

Length of output: 13740


Emit the maximum delay and count thresholded samples.

RelayObservability.flush() emits only eventLoopDelayMsP99. It does not emit eventLoopDelayMsMax, which the new Terraform metric extracts. The metric therefore has no event-loop maximum samples. After adding that field, use a separate INT64 log metric filtered by jsonPayload.eventLoopDelayMsMax >= 5000, with ALIGN_SUM and a threshold above zero. This avoids losing one freeze to the 99th-percentile aggregation.

🐛 Suggested runtime fix
     const memory = process.memoryUsage()
     const p99 = this.eventLoop.count === 0 ? 0 : this.eventLoop.percentile(99) / 1_000_000
+    const max = this.eventLoop.count === 0 ? 0 : this.eventLoop.max / 1_000_000
     this.eventLoop.reset()
...
-      eventLoopDelayMsP99: Number(p99.toFixed(3))
+      eventLoopDelayMsP99: Number(p99.toFixed(3)),
+      eventLoopDelayMsMax: Number(max.toFixed(3))

…ter-based stall alert; cheaper probed sidecar

- deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform
  apply created; the deploy preserves that shape instead of the serving one. It must be the
  latest created, unserved revision on the serving image. Wired into the staging and
  production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE.
- Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000.
- Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU
  with concurrency above 1; checked with validateOnly against the API).
- staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the
  adopted apply shape does not fall to the 80 / 30 s defaults.
@Jinwoo-H Jinwoo-H changed the title fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), gated event-loop stall alert fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), adoptable deploy shape, event-loop stall alert Oct 11, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 418a2f1f-cc1a-4f92-bae6-4b70429f94ab
📥 Commits

Reviewing files that changed from the base of the PR and between 205844f and 9e3f7da.

📒 Files selected for processing (9)
  • .github/workflows/cloud-deploy-relay-production-director.yml
  • .github/workflows/cloud-deploy-relay-staging.yml
  • cloud/dev/fixtures/terraform-root-partition/families.json
  • cloud/dev/scripts/deploy-relay-blue-green.mjs
  • cloud/dev/scripts/deploy-relay-blue-green.test.mjs
  • cloud/infra/terraform/environments/staging.tfvars
  • cloud/infra/terraform/relay-observability.tf
  • cloud/infra/terraform/relay.tf
  • cloud/infra/terraform/variables.tf
🚧 Files skipped from review as they are similar to previous changes (1)
  • cloud/infra/terraform/variables.tf

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

…age changes; GOMAXPROCS=1 on the sidecar

parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a
director topology change or revision pruning, so every caller is covered, and the deploy
refuses a shape baseline whose image differs from the serving one. Both director workflows
also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its
0.25 vCPU quota. The long-hold alert comment is back above its resource.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration
  • Configuration used: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: b5be8c6b-8b94-412f-a6f2-240e8a04b22e
📥 Commits

Reviewing files that changed from the base of the PR and between 9e3f7da and d7b8bb5.

📒 Files selected for processing (6)
  • .github/workflows/cloud-deploy-relay-production-director.yml
  • .github/workflows/cloud-deploy-relay-staging.yml
  • cloud/dev/scripts/deploy-relay-blue-green.mjs
  • cloud/dev/scripts/deploy-relay-blue-green.test.mjs
  • cloud/infra/terraform/relay-observability.tf
  • cloud/infra/terraform/relay.tf
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/cloud-deploy-relay-production-director.yml
  • cloud/infra/terraform/relay-observability.tf

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.

}
const servingImage = servingRevision.spec?.containers?.[0]?.image
// A shape change and an image change never ride together: each is reviewed on its own.
if (config.image?.split('@').at(-1) !== servingImage?.split('@').at(-1)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Require the exact serving image during shape adoption.

If a direct script caller supplies a different registry or image name with the same digest, this check accepts it. deployDirector then passes that different image reference to the rollback and candidate deployments. Compare config.image with servingImage in full, and add a test for matching digests with different image references. The staging workflow's full-image check does not protect direct script calls.

@Jinwoo-H
Jinwoo-H merged commit 2bdb553 into main Oct 11, 2026
22 checks passed
Jinwoo-H added a commit that referenced this pull request Oct 11, 2026
…an deploy it (#27462)

The staging director deploy binds its digest to staging-gce-c4's checked-in image. E is main
2bdb553 (#27405 event-loop max, #27406 sidecar), already mirrored to staging. C4 itself
changes only through its own reviewed refresh workflow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant