Repository navigation
fix(relay): director reaches Cloud SQL through a proxy sidecar (staging), adoptable deploy shape, event-loop stall alert - #27406
Conversation
…ng), gated event-loop stall alert On Cloud Run's built-in /cloudsql socket a connect() blocks the director's whole event loop until the connector's upstream dial ends, 10 s per attempt while the instance starts. Staging froze both directors for 134 s on 2026-10-11 and production four directors for 40-50 s on 2026-09-28. relay_director_cloud_sql_proxy_sidecar runs the Cloud SQL Auth Proxy beside the director on 127.0.0.1:5432, as the GCE cells already do, with its own TCP database URL secret. relay_event_loop_stall_alert adds the event_loop_ms_max metric and a 5 s stall alert. Both default off, so production plans unchanged; staging turns both on.
📝 Walkthrough
Priority: ➖ Normal Merge Risk: 🟡 Moderate · up to The new event-loop stall alert would not fire because the relay runtime does not emit the field the metric filters on. Emit eventLoopDelayMsMax from the runtime before relying on this alert. The sidecar and deploy changes are otherwise bounded. Pre-merge checks |
|
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
28079f63-3e5d-438a-9856-0e07ec443bf7
📒 Files selected for processing (6)
cloud/dev/fixtures/terraform-root-partition/families.jsoncloud/infra/terraform/environments/staging.tfvarscloud/infra/terraform/relay-database.tfcloud/infra/terraform/relay-observability.tfcloud/infra/terraform/relay.tfcloud/infra/terraform/variables.tf
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 4 remain after this review.
| threshold_run = 5000 | ||
| threshold_gce = 5000 | ||
| duration = "0s" | ||
| aligner = "ALIGN_PERCENTILE_99" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🔎 Supported by static analysis
🏁 Script executed:
set -eu
printf '%s\n' '--- diff ---'
git diff --no-ext-diff --unified=80 08c8d30f105f96093ca547d36764e4aa266b92cf 205844f600b8a25c5b0283533da2bc4e4ba716af -- cloud/infra/terraform/relay-observability.tf cloud/infra/terraform | sed -n '1,360p'
printf '%s\n' '--- alert/metric references ---'
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' . || test "$?" -eq 1
printf '%s\n' '--- relevant resource blocks ---'
nl -ba cloud/infra/terraform/relay-observability.tf | sed -n '110,215p'
printf '%s\n' '--- metric-name references ---'
rg -n -F --glob '!**/.git/**' -- 'event_loop_ms_max' . || test "$?" -eq 1Repository: stablyai/orca
Length of output: 37228
🏁 Script executed:
set -eu
git diff --no-ext-diff --unified=80 08c8d30f105f96093ca547d36764e4aa266b92cf 205844f600b8a25c5b0283533da2bc4e4ba716af -- cloud/infra/terraform/relay-observability.tf
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' . || test "$?" -eq 1
nl -ba cloud/infra/terraform/relay-observability.tf | sed -n '110,215p'
rg -n -F --glob '!**/.git/**' -- 'event_loop_ms_max' . || test "$?" -eq 1Repository: stablyai/orca
Length of output: 40632
🏁 Script executed:
set -eu
printf '%s\n' '--- runtime metric wiring ---'
rg -n -F --glob '!**/.git/**' -- 'relay_runtime_log_filter' cloud/infra/terraform
rg -n -F --glob '!**/.git/**' -- '30-second runtime sample' cloud/infra/terraform
rg -n -F --glob '!**/.git/**' -- 'relay_snapshot' cloud/infra/terraform
printf '%s\n' '--- event-loop field variants ---'
rg -n -i --glob '!**/.git/**' -- 'event.?loop|eventLoopDelay|runtime snapshot|runtime metrics' src cloud dev tests 2>/dev/null || test "$?" -eq 1
printf '%s\n' '--- relay rollout limits ---'
rg -n -F --glob '*.tfvars' -- 'relay_max_instances' cloud/infra/terraform
rg -n -F --glob '*.tfvars' -- 'relay_min_instances' cloud/infra/terraformRepository: stablyai/orca
Length of output: 41293
🏁 Script executed:
set -eu
printf '%s\n' '--- relay runtime emitter ---'
nl -ba cloud/apps/relay/src/relay-observability.ts | sed -n '430,615p'
printf '%s\n' '--- flag contract ---'
nl -ba cloud/infra/terraform/variables.tf | sed -n '475,500p'
printf '%s\n' '--- cadence reference ---'
nl -ba cloud/dev/scripts/relay-same-cap-shadow-gate-verdict.mjs | sed -n '135,165p'
printf '%s\n' '--- exact max-field references ---'
rg -n -F --glob '!**/.git/**' -- 'eventLoopDelayMsMax' cloud/apps cloud/dev cloud/infra tests || test "$?" -eq 1Repository: stablyai/orca
Length of output: 13740
Emit the maximum delay and count thresholded samples.
RelayObservability.flush() emits only eventLoopDelayMsP99. It does not emit eventLoopDelayMsMax, which the new Terraform metric extracts. The metric therefore has no event-loop maximum samples. After adding that field, use a separate INT64 log metric filtered by jsonPayload.eventLoopDelayMsMax >= 5000, with ALIGN_SUM and a threshold above zero. This avoids losing one freeze to the 99th-percentile aggregation.
🐛 Suggested runtime fix
const memory = process.memoryUsage()
const p99 = this.eventLoop.count === 0 ? 0 : this.eventLoop.percentile(99) / 1_000_000
+ const max = this.eventLoop.count === 0 ? 0 : this.eventLoop.max / 1_000_000
this.eventLoop.reset()
...
- eventLoopDelayMsP99: Number(p99.toFixed(3))
+ eventLoopDelayMsP99: Number(p99.toFixed(3)),
+ eventLoopDelayMsMax: Number(max.toFixed(3))…ter-based stall alert; cheaper probed sidecar - deploy-relay-blue-green: --shape-baseline-revision names the 0% revision a Terraform apply created; the deploy preserves that shape instead of the serving one. It must be the latest created, unserved revision on the serving image. Wired into the staging and production director workflows behind ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE. - Event-loop stall alert is now a gated counter metric on eventLoopDelayMsMax>=5000. - Sidecar gets a /liveness probe and 0.25 vCPU (Cloud Run rejects only a total under 1 vCPU with concurrency above 1; checked with validateOnly against the API). - staging.tfvars pins the serving director's concurrency 1000 and timeout 3600 s, so the adopted apply shape does not fall to the 80 / 30 s defaults.
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
418a2f1f-cc1a-4f92-bae6-4b70429f94ab
📒 Files selected for processing (9)
.github/workflows/cloud-deploy-relay-production-director.yml.github/workflows/cloud-deploy-relay-staging.ymlcloud/dev/fixtures/terraform-root-partition/families.jsoncloud/dev/scripts/deploy-relay-blue-green.mjscloud/dev/scripts/deploy-relay-blue-green.test.mjscloud/infra/terraform/environments/staging.tfvarscloud/infra/terraform/relay-observability.tfcloud/infra/terraform/relay.tfcloud/infra/terraform/variables.tf
🚧 Files skipped from review as they are similar to previous changes (1)
- cloud/infra/terraform/variables.tf
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.
…age changes; GOMAXPROCS=1 on the sidecar parseArguments now refuses --shape-baseline-revision with a runtime identity bootstrap, a director topology change or revision pruning, so every caller is covered, and the deploy refuses a shape baseline whose image differs from the serving one. Both director workflows also refuse the combinations up front. The proxy sidecar runs with GOMAXPROCS=1 to match its 0.25 vCPU quota. The long-hold alert comment is back above its resource.
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository UI
- Review profile: CHILL
- Plan: Advanced
- Run ID:
b5be8c6b-8b94-412f-a6f2-240e8a04b22e
📒 Files selected for processing (6)
.github/workflows/cloud-deploy-relay-production-director.yml.github/workflows/cloud-deploy-relay-staging.ymlcloud/dev/scripts/deploy-relay-blue-green.mjscloud/dev/scripts/deploy-relay-blue-green.test.mjscloud/infra/terraform/relay-observability.tfcloud/infra/terraform/relay.tf
🚧 Files skipped from review as they are similar to previous changes (2)
- .github/workflows/cloud-deploy-relay-production-director.yml
- cloud/infra/terraform/relay-observability.tf
Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 3 remain after this review.
| } | ||
| const servingImage = servingRevision.spec?.containers?.[0]?.image | ||
| // A shape change and an image change never ride together: each is reviewed on its own. | ||
| if (config.image?.split('@').at(-1) !== servingImage?.split('@').at(-1)) { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Require the exact serving image during shape adoption.
If a direct script caller supplies a different registry or image name with the same digest, this check accepts it. deployDirector then passes that different image reference to the rollback and candidate deployments. Compare config.image with servingImage in full, and add a test for matching digests with different image references. The staging workflow's full-image check does not protect direct script calls.
ELI5
While its database is starting up or failing over, the relay director freezes. Every attempt to reach the database stops the whole process for up to 10 seconds, and the attempts follow each other back to back. This PR gives the director the same small database proxy the relay cells already run beside them. With it, a database outage makes the director answer "try again" within a couple of seconds instead of going silent.
It's switched on in staging only. Production plans don't change until we flip it there, and the rollout steps for both are written out below.
What Changed
Two new Terraform variables, both off by default.
staging.tfvarsturns both on.relay_director_cloud_sql_proxy_sidecar. The director's Cloud Run template gains acloud-sql-proxysidecar.relay_gce_cloud_sql_proxy_image(v2.18.2), and listens on127.0.0.1:5432. The cells run the same proxy beside the relay, but over a Unix socket on a full Linux kernel./startupstartup probe, and a/livenessliveness probe every 10 s with 3 failures allowed.--help,/startupdoesn't wait for the database, so the director still boots during an outage./livenessfails only when the proxy itself is wedged.relay, stayscontainers[0], anddepends_onthe proxy./cloudsqlvolume and its mount are dropped.DATABASE_URLreads a new secret,orca-cloud-relay-director-database-url(postgresql://…@127.0.0.1:5432/orca_relay), readable by the director runtime account. Cells keep the socket URL secret.roles/cloudsql.client(relay.tf:23). It dials the public IP, as today's built-in connector does.GOMAXPROCS=1. Go would otherwise size its scheduler to the host's cores rather than the 0.25 vCPU quota, and get throttled in bursts.validateOnly: relay 1 vCPU plus proxy 0.25/0.5/0.08 with concurrency 1000 is accepted, and a relay at 0.5 vCPU is rejected with "Total cpu < 1 is not supported with concurrency > 1". So director compute goes up 25%, not 100%.relay_event_loop_stall_alert. Adds a counter metric,orca_relay_event_loop_stalls, that counts runtime samples withjsonPayload.eventLoopDelayMsMax>=5000, plus anOrca Relay: event loop stallalert on any count. It follows therelay_long_cell_inventory_holdpattern: the filter does the threshold, so any value is a breach. There is no percentile over a max. The alert doesn't page yet. The field comes from fix(relay): report event-loop freezes and keep seat polls off the cell-list read #27405.A reviewed one-time shape transition in the deploy script.
assertPreservedRevisionShapecompares every candidate with the serving revision, and the sidecar, the dropped volume, the secret reference and the container name all differ.deploy-relay-blue-green.mjs --shape-baseline-revision <name>makes the guard compare against the named revision instead.--imagedigest equal to the serving digest). A shape change and an image change never ride together.parseArgumentsrefuses to combine it with--bootstrap-runtime-identity true, a topology change (--capacity-cell-idor--director-cells-json) or--prune-revisions true. That covers every caller, not just the workflows.serving.knative.dev/creatorannotation names the principal, not the tool. In staging, deploy-script revisions and workflow Terraform applies both run asorca-cloud-staging-gha-relay, and operatorgcloudrevisions showjinwoo@stably.aiwithclient-name: gcloud. A creator check would either block a legitimate workflow apply or pass a hand-made revision. The latest-created, unserved and same-image checks bound the choice instead.shape-baseline-revisioninput. It is validated against the service's revision-name pattern, needs the confirmationADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE, and refuses to combine with bootstrap, prune or a regional-placement change. When the input is empty, both workflows behave exactly as before.Staging drift pinned.
staging.tfvarspins both to what is serving, so the apply leaves them as they are.The five new resource families are registered in the Terraform root-partition fixture.
Why
Root cause. Cloud Run's first-generation built-in Cloud SQL connection embeds the Auth Proxy behind the
/cloudsqlsocket. On that socket,connect()itself blocks until the proxy's dial to the instance ends, and while an instance is starting that dial times out after 10 s on each try. The director's awaits run back to back without returning to the event loop, so no timer fires. That covers the pool's 2 s connect timeout, the 4 s assign queue wait, metrics, and the seat polls that keep reserve cells fed.What it cost.
Why the sidecar. It accepts a loopback TCP connection straight away and dials in the background. The pool's 2 s timeout then works, and a failed attempt costs the loop nothing.
Alternatives considered.
@google-cloud/cloud-sql-connector. A new dependency, with TLS handled in the relay.PUBLIC_STICKY_*,PUBLIC_ASSIGNMENTS_ENABLED, and emptyFENCE/MONITORservice-account variables; production adds an emptyASIA_PROOF_SERVICE_ACCOUNT. Each of those also fails the guard. Naming the reviewed apply revision covers the whole reviewed plan, and nothing outside it.Staging enablement sequence (after merge, with Jinwoo's go)
Wake staging if it's asleep (
Power Relay Staging,wake,wake-cells: all). Confirm Cloud SQL isRUNNABLE.Record the pre-state:
gcloud run services describe orca-cloud-relay-staging --project onorca-cloud-staging --region us-central1)ORCA_RELAY_CELLS_JSON,ORCA_RELAY_RESERVE_PLACEMENTandORCA_RELAY_SHADOW_SEAT_FEED_CELLSPlan and review. From
cloud/infra/terraform, with aGOOGLE_OAUTH_ACCESS_TOKENfromgcloud auth print-access-token, run a saved targeted plan:Expect
Plan: 5 to add, 1 to change, 0 to destroy. The read-only plan from this branch gives exactly that.relayname anddepends_onDATABASE_URLsecret swap, and the/cloudsqlvolume and mount droppedORCA_RELAY_IMAGE_DIGESTremovedORCA_RELAY_CELLS_JSONre-ordered with the same valuePUBLIC_STICKY_*andPUBLIC_ASSIGNMENTS_ENABLEDat their app defaultsFENCE/MONITORservice accountsrevisionfieldApply the saved plan with the state lock:
terraform apply director-sidecar.tfplan.Record the 0% revision. Corrected after the staging run (2026-10-11, see docs(relay): director Cloud SQL sidecar enablement and rollback #27461):
status.latestCreatedRevisionNameis a new revision, N, and traffic is still 100% on the step-2 revision.selector-rollbackcandidate in step 6, health-checked before traffic moves.404 Cannot find metric(s): the new metric takes minutes to appear. If so, re-plan and apply the policy alone a few minutes later.Dispatch
Deploy Relay Stagingwith:expected-image-digest= the step-2 digestreserve-placement: preserveexpected-rehome-generation= the step-2 valuebootstrap-runtime-identity: falsecapacity-cell-id: nonemin-instances: preserveshadow-seat-feed-cells: preserveshape-baseline-revision: Nconfirmation: ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPEVerify the new serving revision:
relayfirstDATABASE_URLset toorca-cloud-relay-director-database-url:latestORCA_RELAY_IMAGE_DIGESTback, minScale 2, and the step-2 reserve and shadow settings unchangedcloudsql-instancesannotation127.0.0.1:5432, the director reaches/ready, there are no neworca_relay_postgres_query_failedlines, and seat summaries and placement summaries keep flowingRun the Cloud SQL stop→start drill. Pass means:
runtime_metricsflush keeps its schedule503 reserve-databasewithin about 5 s while the database startseventLoopDelayMsMaxstays under 1 sDrill measurement, sidecar CPU. Before step 4, run the same assign burst (the staging load workflow's assign phase, same profile both times) on the socket revision, and again after step 7 on the sidecar revision. Record from
runtime_metrics:sql_latency_msp95 and maxdatabasePoolWaitMsMaxAlso record the
cloud-sql-proxycontainer's CPU fromrun.googleapis.com/container/cpu/usagewithmetric.label.container_name="cloud-sql-proxy", as a fraction of its 0.25 vCPU limit. That is the per-container metric;container/cpu/utilizationscarries no container label. Rule: if proxy utilization nears its 0.25 limit (p99 above about 80%), or SQL latency p95 rises past the socket run's, raise the sidecar to 0.5 vCPU in a follow-up before production.Rollback at any point after step 6. Re-dispatch the deploy with the step-2 digest after a reviewed apply with the variable set to
false, again usingshape-baseline-revisionfor the apply's revision. The step-2 revision and the socket secret are untouched throughout.Emergency socket rollback. If the sidecar revision misbehaves and the reviewed rollback above is too slow, move traffic straight back to the pre-sidecar revision. That revision survives the deploy because the shape-adoption deploy refuses
--prune-revisions, and it still carries its own/cloudsqlconnection and socket URL secret:No repository workflow owns an arbitrary-revision traffic move today. The deploy script only moves traffic to its own tagged candidate or rollback. So this is an operator action with
run.admin: in production it needs Jinwoo's go and the production rollout lease. Afterwards the service template is still the sidecar shape. The next deploy refuses (serving revision on the socket shape, template on the sidecar shape) until a reviewed apply with the variable set tofalseand a deploy that adopts that apply's revision.Production enablement path (not run; for review)
relay_director_cloud_sql_proxy_sidecar = true(and the alert variable) inproduction.tfvars, after the staging drill passes.Plan: 5 to add, 1 to change, 0 to destroy, with the same targets as staging step 3.ORCA_RELAY_IMAGE_DIGESTremoved (the deploy restores it)ORCA_RELAY_CELLS_JSONre-ordered with the same valueORCA_RELAY_ASIA_PROOF_SERVICE_ACCOUNTaddedrevisionand service-levelscalingfields clearedDeploy Relay Production Director. Use the serving digest,bootstrap-runtime-identity: false,prune-incompatible-revisions: false,regional-placement-mode: preserve,shape-baseline-revision: <apply revision>andconfirmation: ADOPT_RELAY_DIRECTOR_TERRAFORM_SHAPE. The deploy is blue/green and keeps the serving floor, so nothing disconnects.Orca Relay: event loop stalland the 30 s flush cadence across the next Cloud SQL maintenance window.Linked Issue
N/A (internal relay incident follow-up).
Visual Proof
N/A: infrastructure only.
Testing
terraform fmt -check -recursiveandterraform validatepass.-lock=falseplans, refreshed). Targeted at every resource this PR touches, branch andmaingive an identical plan:Plan: 10 to add, 2 to change, 0 to destroy, with the same action list and the same rendered diff. This PR adds 0 production changes.connection_headroomalert, and the director's deploy-owned fields.0 to add, 1 to change, and that one change is the same existing director drift.-var, narrow targets):5 to add, 1 to change, as listed under the production path.5 to add, 1 to change. The director update no longer touches concurrency or timeout.validateOnlyon staging (nothing persisted):/livenessprobe anddepends_on.deploy-relay-blue-green.test.mjs: 50 pass. New cases:relay-load-control-peer, needs an unbuilt package in this lane and passes where the packages are built.actionlinton both changed workflows reports nothing new; it only flags the existingblacksmithrunner label.AI Disclosure
N/A (internal contributor).
Agent skill upstream boundary
Notes
/cloudsqlannotation and dependency annotation names come from the live revision's v1 shape. The deploy no longer compares them across the transition, so a naming surprise can't wedge it; step 7 confirms what Cloud Run actually wrote.Checklist