Repository navigation
Conversation
…eiling The shared Cloud SQL connection budget has been reporting green on two wrong numbers. auth was recorded at 2 instances. Production raised auth_max_instances from 2 to 20 by hand on 2026-09-04, after a cap of 2 starved desktop token refresh with hours of 429s. The private orca-cloud copy of this contract has said 20 since then; this public copy never followed. The ceiling was recorded as 400, described as the tier default. No max_connections flag is set, so it is the tier default, but the tier default for db-custom-4-15360 is not 400. SHOW max_connections on the live instance returns 500. With both corrected the budget no longer fits: 495 configured plus a 215-connection rollout overlap and a 5-connection admin allowance is an operating maximum of 715 against a usable ceiling of 490. auth is 415 of that 715. The single withinBudget assertion becomes two named gates so the failure says which bound broke, and both carry the full arithmetic in their message. Nothing here changes a pool, an instance cap, or a reserve. Closing the gap is an auth sizing decision; the PR body lays out the options and what each one costs.
This was referenced Sep 17, 2026
Merged
Jinwoo-H
added a commit
that referenced
this pull request
Sep 17, 2026
The budget's usable ceiling came from maxConnections: 400, described as the tier default. It is a tier default, since no max_connections flag is set, but the instance does not report 400. SHOW max_connections on it returns 500, measured 2026-09-16. On main the model sat at 385 against a usable ceiling of 390, five connections of margin, so raising the three asia-east2 pools by 18 failed the gate by 13 against a ceiling that was never checked. Against the measured one it is 403 against 490, clearing by 87. Only the ceiling and its source note change here. auth stays recorded at 2 instances, which is also wrong; PR #21165 corrects it, and with the true auth figure the budget is over by 225 for reasons that have nothing to do with these pools.
Jinwoo-H
added a commit
that referenced
this pull request
Sep 17, 2026
…ed connection ceiling (#21163) * infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells The three asia-east2 cells sit 176 ms from the Cloud SQL instance in us-central1. Server-side statement time there is 0.2 ms, so a pool slot is held by the round trip, not by the query. At a pool of 10 they measured 94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against 222-646 ms in us-central1. Raising those three pools to 16 is the agreed first step; every other cell stays at 10. c4 and c5 join the committed fence set. Both are existing-only capacity the admission selector can never place on again, they carried ~1 connection each on 40-day-old images, and each still holds 10 Postgres connections. The fence set is the prerequisite the fence-source workflow confirms before it drains and attests a cell; it is not itself the resize. c17 and c18 are not fenced here. They are migration-only, and the runbook requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector CAS before it can be fenced. Terraform cannot express that step. The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances when production has run a cap of 20 since 2026-09-04, and a 400-connection ceiling when the live instance reports 500. Both are corrected, and the budget now asserts its headroom in two named gates instead of one aggregate boolean. Those gates fail: auth alone accounts for 200 configured connections and a 215-connection rollout overlap, so the operating maximum is 713 against a usable ceiling of 490. Nothing here caused that, and no pool was lowered to hide it. * infra(relay): move the Cloud SQL contract correction out of this branch The contract correction (auth at its real 20-instance cap, the measured 500-connection ceiling) makes the budget gate fail for reasons that have nothing to do with asia pools or fenced cells, and it held this branch red. It moves to its own branch where the failure is the subject. production-cloud-sql-app-consumers.json returns to main unchanged. The budget test keeps main's single gate and only repins the cell figure that this branch genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for fencing c4 and c5. Against main's 400-connection model that leaves an operating maximum of 383 under a usable ceiling of 390. * infra(relay): move the c4/c5 fence entries out of this branch Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells (relay-gce-cells.tf); the lifecycle ignore that used to protect operational target_size drift is gone. So a fence entry sitting on main ahead of its fence-source run is a standing instruction that any apply reaching that cell may execute without the documented drain and attestation. Keeping the entry in the same merge as an unrelated pool change widens that blast radius for no reason. The two entries move to their own branch, to be merged immediately before fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of the list, which makes that later diff two added lines instead of a rewritten one. The cell figure in the budget test follows: 230 + 18 for the three asia-east2 pools at 16, with no fenced-cell subtraction. That is 403 operating against a usable ceiling of 390, so the headroom gate now fails by 13. It fails against a ceiling of 400 that is itself wrong; the instance reports 500. See the PR body. * infra(cloud-sql): record the measured 500-connection ceiling The budget's usable ceiling came from maxConnections: 400, described as the tier default. It is a tier default, since no max_connections flag is set, but the instance does not report 400. SHOW max_connections on it returns 500, measured 2026-09-16. On main the model sat at 385 against a usable ceiling of 390, five connections of margin, so raising the three asia-east2 pools by 18 failed the gate by 13 against a ceiling that was never checked. Against the measured one it is 403 against 490, clearing by 87. Only the ceiling and its source note change here. auth stays recorded at 2 instances, which is also wrong; PR #21165 corrects it, and with the true auth figure the budget is over by 225 for reasons that have nothing to do with these pools. * test(cloud): state the cell pool arithmetic literally in the budget pin comment
…et-truth # Conflicts: # cloud/dev/contracts/production-cloud-sql-app-consumers.json # cloud/dev/scripts/relay-cloud-sql-connection-budget.test.mjs
… instance floor Two corrections, in opposite directions. auth was recorded at 2 instances. orca-cloud PR #474 raised auth_max_instances 2 -> 20 on 2026-09-04, after a cap of 2 starved desktop token refresh for hours. Production has since used it: a 14-day pull shows instance maxima of 21, 16 and 18, with per-database num_backends of 205, 165 and 114 on those days, 9.2-9.8 connections per instance. The cap is not headroom nobody touches, so the honest figure is 200 and auth is not being cut to make a gate pass. The rollout overlap was too pessimistic in the other direction. It billed a candidate revision at its full instance ceiling, on the premise that a tagged revision escapes the service cap. The auth and API deploy scripts both build the candidate with --tag and --no-traffic and never pass --min, then promote with an atomic traffic flip, so each candidate holds only its service floor until the flip: one instance for auth, and for API a service floor of zero lifted to one by the smoke run against the tag origin. Those terms become 25 and 20. The director candidate is the exception and keeps its full term. Its workflow deliberately omits --min-instances so the candidate inherits the Terraform floor of five and genuinely dual-serves; pinning it to one previously rebuilt a per-instance admission shortage. At 30 it is now the widest overlap. Net: 513 configured, 548 operating, against a usable ceiling of 490. Still over, by 58. Every option that closes it on paper removes capacity production has already used, so nothing here lowers a pool, an instance cap, or the reserve. The gate stays red until the instance itself grows.
Jinwoo-H
changed the title
9 tasks done
ivanvykopal
pushed a commit
to ivanvykopal/orca
that referenced
this pull request
Sep 20, 2026
…ed connection ceiling (stablyai#21163) * infra(relay): raise asia-east2 cell pools to 16 and retire four idle cells The three asia-east2 cells sit 176 ms from the Cloud SQL instance in us-central1. Server-side statement time there is 0.2 ms, so a pool slot is held by the round trip, not by the query. At a pool of 10 they measured 94-156 waiters and 2 s waits, and client accepts ran a ~4 s p95 against 222-646 ms in us-central1. Raising those three pools to 16 is the agreed first step; every other cell stays at 10. c4 and c5 join the committed fence set. Both are existing-only capacity the admission selector can never place on again, they carried ~1 connection each on 40-day-old images, and each still holds 10 Postgres connections. The fence set is the prerequisite the fence-source workflow confirms before it drains and attests a cell; it is not itself the resize. c17 and c18 are not fenced here. They are migration-only, and the runbook requires retire-migration-cell to move a migration-only cell to existing-only through a generation-bound selector CAS before it can be fenced. Terraform cannot express that step. The Cloud SQL consumer contract carried two stale numbers: auth at 2 instances when production has run a cap of 20 since 2026-09-04, and a 400-connection ceiling when the live instance reports 500. Both are corrected, and the budget now asserts its headroom in two named gates instead of one aggregate boolean. Those gates fail: auth alone accounts for 200 configured connections and a 215-connection rollout overlap, so the operating maximum is 713 against a usable ceiling of 490. Nothing here caused that, and no pool was lowered to hide it. * infra(relay): move the Cloud SQL contract correction out of this branch The contract correction (auth at its real 20-instance cap, the measured 500-connection ceiling) makes the budget gate fail for reasons that have nothing to do with asia pools or fenced cells, and it held this branch red. It moves to its own branch where the failure is the subject. production-cloud-sql-app-consumers.json returns to main unchanged. The budget test keeps main's single gate and only repins the cell figure that this branch genuinely moves: 230 -> 228, being +18 for three asia pools at 16 and -20 for fencing c4 and c5. Against main's 400-connection model that leaves an operating maximum of 383 under a usable ceiling of 390. * infra(relay): move the c4/c5 fence entries out of this branch Terraform now sets a cell's MIG target size directly from relay_gce_fenced_cells (relay-gce-cells.tf); the lifecycle ignore that used to protect operational target_size drift is gone. So a fence entry sitting on main ahead of its fence-source run is a standing instruction that any apply reaching that cell may execute without the documented drain and attestation. Keeping the entry in the same merge as an unrelated pool change widens that blast radius for no reason. The two entries move to their own branch, to be merged immediately before fence-source runs for c4 and then c5. This branch keeps the multi-line reflow of the list, which makes that later diff two added lines instead of a rewritten one. The cell figure in the budget test follows: 230 + 18 for the three asia-east2 pools at 16, with no fenced-cell subtraction. That is 403 operating against a usable ceiling of 390, so the headroom gate now fails by 13. It fails against a ceiling of 400 that is itself wrong; the instance reports 500. See the PR body. * infra(cloud-sql): record the measured 500-connection ceiling The budget's usable ceiling came from maxConnections: 400, described as the tier default. It is a tier default, since no max_connections flag is set, but the instance does not report 400. SHOW max_connections on it returns 500, measured 2026-09-16. On main the model sat at 385 against a usable ceiling of 390, five connections of margin, so raising the three asia-east2 pools by 18 failed the gate by 13 against a ceiling that was never checked. Against the measured one it is 403 against 490, clearing by 87. Only the ceiling and its source note change here. auth stays recorded at 2 instances, which is also wrong; PR stablyai#21165 corrects it, and with the true auth figure the budget is over by 225 for reasons that have nothing to do with these pools. * test(cloud): state the cell pool arithmetic literally in the budget pin comment
Collaborator
|
Closing because #31792 ( This PR corrects auth instance capacity and tagged-candidate overlap in the shared Cloud SQL budget/contract. All four target contract/operator/test files were removed from this public repository by that migration. The authoritative budget and any capacity correction now belong beside the private deployment configuration. Closure does not assert that the 58-connection worst-case shortfall was fixed, that the database was resized, or that private/live capacity has been verified. I reviewed the complete patch and discussion. The private repository is inaccessible to this audit, so any still-needed work must be carried forward there; this is a repository-ownership closure, not a claim that the operational issue is fixed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ELI5
We keep a tally of how many database connections everything can ask for at once, so we notice before we run out. Two entries in it were wrong in opposite directions, and fixing both still leaves us over the limit.
The failing test is the point of this PR. The honest answer is that the database is too small for what production already runs, and the fix is a bigger database, not smaller services.
What Changed
The login service's real size.
authInstancesgoes from 2 to 20. The cap was raised to 20 on 2026-09-04 in orca-cloud PR #474, after a cap of 2 starved desktop token refresh for hours. It is not spare headroom: a 14-day Monitoring pull shows instance maxima of 21, 16 and 18 on three separate days, with per-databasenum_backendsof 205, 165 and 114 on those days. That is 9.2 to 9.8 connections per instance against a pool of 10, so the pool does fill during refresh storms.How a deployment's temporary second copy is billed. The model charged a candidate revision at its full instance ceiling, on the premise that a tagged revision escapes the service-wide cap. That is not how auth and API deploy. Both build the candidate with
--tagand--no-traffic, never pass--min, and promote with an atomic traffic flip, so the candidate holds only its service floor until the flip. Auth's floor is 1. API's is 0, lifted to 1 by the serial smoke run against the tag origin before promotion. Those overlap terms drop from 215 and 65 to 25 and 20.The director candidate keeps its full term. Its workflow deliberately omits
--min-instancesso the candidate inherits the Terraform-owned floor of 5 and genuinely dual-serves. A comment there records that pinning it to 1 previously rebuilt a per-instance admission shortage that took placement failures to about 70 percent. At 30 it is now the widest overlap, which it should be.Two new contract fields,
authCandidateMinInstancesandapiCandidateMinInstances, carry the floors with source notes. The guard test that pinned the old premise is rewritten to pin the new one, including an assertion that the director's blue/green call passes no--min-instances.Why
The decision this PR encodes is that auth is not cut. Its pool stays at 10 and its instance cap stays at 20.
Every option that makes the budget fit on paper removes capacity production has already used. Halving the pool to 5 would cap auth at its observed per-instance draw during the exact events the cap was raised for. Halving the instance cap to 10 would put it below two of the three observed maxima. Neither is a spare-capacity trim; both are a rollback of an incident fix, and the incident they would re-open is hours of 429s on desktop token refresh.
So the budget is corrected to say what is true and left failing. The alternative was to shrink a pool or the reserve until the number fit, which would make a real capacity signal quieter rather than more accurate.
Linked Issue
Fixes #
Visual Proof
N/A— a data contract, a calculation, and their tests. No user interface surface.Testing
Run from
cloud/.pnpm -r testnode --testscript listpretestscript listterraform fmt -check -recursiveThe overshoot is 58, not the ~75 estimated when this rework was scoped. Two reasons. The estimate was taken against 230 relay cells, and #21163 has since merged and raised three asia-east2 pools, so the figure is 248. And remodelling the API candidate as well as the auth one dropped the widest overlap from 65 to 30, which gave back 35.
Relay cells are 248 of the 548. No relay-side change closes a 58-connection gap, and none is attempted here.
AI Disclosure
Review
What actually closes this
The instance has to grow. Moving Cloud SQL to
db-custom-8-30720withmax_connections=750clears 548 against a usable 740 with 192 to spare.That is a restarting change and needs the owner's maintenance window. It is justified on its own terms regardless of this budget: memory on the current
db-custom-4-15360runs at 78 to 86 percent daily, so the tier is already the constraint rather than the connection count.The live control until then
The budget is a worst-case model, not a measurement. The thing that would actually catch a real exhaustion is the backend alerting: a freeze at 250 and a page at 320, against an observed 14-day peak of 279 out of 500. That peak is the number to watch, and it sits between the two thresholds today.
Read this PR's red gate as "the worst case no longer fits", not as "the instance is currently out of connections".
Where the evidence came from, line by line
--tagand--no-trafficwith--maxand no--min, and promotion is a single--to-tags=<tag>=100. The service floor isauth_min_instances = 1. Corroborated by an observation on 2026-09-17 of four revisions with live instances totalling 165 backends.--tagand--no-trafficwith no--min, against a servicemin_instances = 0. Its deploy smoke-tests the candidate through the tag origin before--to-revisions=<candidate>=100, so one instance is the honest pre-flip figure rather than zero.cloud-deploy-relay-production-director.ymlpasses--max-instancesand no--min-instances, with a comment stating that the candidate inherits the Terraform-owned scaling and that pinning 1 rebuilt the admission shortage.relay_min_instances = 5.Companion change in the private repo, not made here
stablyai/orca-cloudholds the authoritative copy of this contract and still readsmaxConnections: 400. Its guard test also asserts that nomax_connectionsdatabase flag exists indatabase.tfor either environment's tfvars. Both have to change in the orca-cloud PR that does the tier move, since raising the ceiling means adding exactly the flag that test currently forbids. Nothing in this PR touches that repository.Related
maxConnectionsto the measured 500. Merged, and the reason the cell figure here is 248.Agent skill upstream boundary
docs/reference/agent-skill-sharing-upstream-boundary.mdand copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.Notes
Security: no identity, permission, or trust boundary change. Cross-platform: JSON and Node test files, no platform-dependent behavior. Remote SSH and Mobile: unaffected; the push gateway runs on a separate instance and is not in this budget. Backwards compatibility: no wire format, API, or client contract changes. Performance: this PR changes no running configuration. The condition it documents is a worst-case saturation risk, not a current regression.
Checklist
N/Awith reasonpnpm lint,pnpm typecheck,pnpm test, andpnpm buildpass (or CI will cover; local preferred)