[Jinwoo-H]

	Files	Added	Deleted	Net
Test	4	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​860	$\color{#cf222e}{\Huge{\mathbf{−}}}$​3	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​857
Prod	5	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​1105	$\color{#cf222e}{\Huge{\mathbf{−}}}$​8	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​1097

ELI5

This is the second half of the database move tool. It does the actual switch. It tells every relay server to pause its database work for a few seconds, locks the old database against the relay, copies the last changes, then tells the servers the new database is in charge. If anything goes wrong before that last step, it unlocks the old database and the servers carry on there. Nothing runs until an operator starts it.

What Changed

For users: nothing until an operator runs cutover. During a cutover, relay requests wait for a few seconds and then continue; none should fail.

How it works (the contract shared with #29405):

• fence --side. The tool logs in as the relay role and changes that role's own password to a random one, then ends every other session of that role. It repeats until two checks in a row find none.
  • A role may change its own password, so this needs no admin right.
  • Sign-in's and the API's sessions are never touched.
• restore-password --side. Sets the relay role's password back from that side's secret, through the Cloud SQL Admin API (users.update). This is the fallback when the fencing session is gone. It refuses a side whose generation is below the pointer's; a rollback cutover (--to primary) reopens its own fenced target before the window instead.
• cutover --to dedicated|primary [--dry-run].
  • It first checks:
    • tracking is on both sides and the copy finished;
    • the backlog is small;
    • every relay session carries the mv1 mark (the tool lists any that don't);
    • no other login role can write relay rows or switch to the relay role;
    • the pointer exists.
  • Then it runs these steps:
    1. Writes a pause window into the pointer, about 14 s ahead, with a compare-and-swap on the object generation.
    2. From the window's start, waits until every relay process seen on the source ( mv1:<id>) also holds a session on the target, aborting and naming any still missing 5 s in. It then waits until the source's relay sessions have no running statement or open transaction on two scans 100 ms apart (capped at 6 s, logging what was busy). Then it fences the source while keeping its own relay-role session open.
    3. Copies the remaining changes until none are left.
    4. Compares the small tables in full, and checks once more for a late login.
    5. Writes authority generation + 1 on the target, then records that generation in the pointer.
    6. Clears the window.
  • The processes switch when they read the higher generation.
  • Abort. Any failure before step 5, or past 15 s into the window, or a SIGTERM, restores the source password over the held session and clears the window. If the reply to step 5 is lost, the tool undoes only after a locked re-read proves the write did not land.
  • --dry-run runs every check and changes nothing.
• status. Shows the backlog, the sentinel, the generations, the mv1 coverage and the pointer.

Why

• The fence works without a superuser, which the relay role is not. Losing contact is never treated as a switch signal. The only signals are the generation number and a refused password.
• The password is sent as a SCRAM verifier, not plaintext, so a server with log_statement = ddl never logs it.
• Why a pause rather than a cutover with errors: held requests wait instead of failing, and the 20 s cap is under the 45 s heartbeat staleness.

Stacked on part 1 (#29462).

Ending a move: finish --keep primary|dedicated. It runs only when the kept side is the one serving: the highest generation, not below the pointer's, with the relay login working.

• --keep primary:
  1. Sets the other side's generation to 0.
  2. Sets the pointer's generation floor to 0 and waits two polls.
  3. Drops primary's authority and move tables.
  4. Deletes the pointer.
     Primary stays eligible at every step, and afterwards processes serve DATABASE_URL alone, as before the move.
• --keep dedicated removes only tracking and the move bookkeeping. The pointer and dedicated's authority stay, because they are how processes find dedicated until DATABASE_URL names it.
• Either way, then drop the other database through the Cloud SQL API (gcloud sql databases delete), which needs no login. Dropping it ends the move on that side.
• The side a cutover left is fenced, so uninstall-tracking there refuses (side_is_fenced) and restore-password is refused by the generation guard. Both are by design.

Staging rehearsal (an Enterprise instance stood in for Enterprise Plus, which was out of capacity in us-central1). Four cutovers under load: primary→dedicated during the ramp, dedicated→primary at saturation, primary→dedicated at steady state, dedicated→primary during the ramp.

• Each had 0 keys left to drain at the fence and 0 hot-table mismatches. Processes held for 0.8–1.0 s (one run 1.5 s).
• In the last run's 3-minute window, all 1,365 /v1/assign calls returned 200, including 58 during the hold, with a max latency of 0.34 s. Cells had 0 SQL failures.
• One refusal with no effect: an IAM condition on users.update returned 403, so the cutover refused at reopen-target before writing anything.
• The rehearsal found one bug: a copy made before tracking was reinstalled still counted as done. It is fixed in part 1, and this cutover now refuses such a copy.

Linked Issue

Fixes #

Visual Proof

N/A: an operator command-line tool, no UI.

Testing

Local Postgres 17 and 16:

• Fence:

  • It works for a role with no CREATEROLE or superuser.
  • A new login then gets 28P01, and an existing session keeps working until it is ended.
  • The held session survives.
  • Restoring the original password over the held session lets the role log in again.

• End-to-end cutover:

  • Setup: two relay processes run on feat(relay): follow a database-target pointer with a hold-and-switch between two Postgres targets #29405's SwitchingRelayDatabase, against a fake pointer store, with 6 writers doing relay-shaped work (audit inserts, counter updates, rate-window upserts). Timings are scaled down.
  • Result: 0 failed operations, both processes on the dedicated side, the pause held about 1.25 s (it waited out a 1.2 s transaction that was running when the window opened), and 6-7 relay sessions ended. The pointer records generation 1, and restore-password then refuses primary.
  • The rollback cutover back to primary (it reopens primary itself) moves a fresh fleet with 0 failed operations to generation 2.
  • A process that cannot log in to the target makes the cutover abort and name it, with 0 failed operations and the source untouched.
  • Every acknowledged write is on the dedicated copy: about 850 events, and per-cell counters equal to the acknowledged increments.
  • A reverse sync then makes the copies match exactly (verify).
  • A forward copy into the now-active side is refused.

• Refusals:

  • an unmarked relay session (listed by name, also as seen by the tool role);
  • a role with its own write grant;
  • a NOINHERIT member that can SET ROLE to the relay role.
  • A dry run leaves the pointer unchanged.

• Abort: a row only the target has makes the comparison fail. The source password is restored, the window is cleared, the processes stay on the source, writes continue with 0 failures, and the target generation stays at 0.

• Cloud SQL Admin restore and the GCS pointer store are tested with fake fetches: the URLs, operation polling, 412 on a generation mismatch, and errors that never contain the password.

• I manually tested these changes locally

• Automated tests added/updated, or explained why not below

Review

Agent skill upstream boundary

• Not applicable, or this change copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Still to prove on staging before the first production use:

• the fence, done by a Cloud SQL API-created user;
• what the other-writer check reports for Cloud SQL's built-in roles;
• that a pre-hashed SCRAM password is accepted under any password-policy flags.

Checklist

• This PR is small and focused
• I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
• Before/after screenshots or videos attached for UI changes, or N/A with reason
• Self-reviewed for correctness, security, and performance
• Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
• pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

[OrcaWin]

Closing this public-repository PR because its core is deployment operator tooling, which moved to private stablyai/orca-cloud in merged #31792 (307f0cdaeed). The current cloud README explicitly places “operator tools” there while keeping runtime and wire contracts public: https://github.com/stablyai/orca/blob/1a723194c3294298c8db106105d1be226823ae9c/cloud/README.md#deployment-configuration

I reviewed the full body, production file scope, CLI entry point/import graph and regression intent. This PR provides operator-invoked fencing, password restoration, cutover, status and finish. It is not called by the relay serving path. The new production modules are confined to the database-move CLI graph; this slice changes only the CLI and its password/cutover/fence modules.

The tool's migration value survives, so the code and its Postgres race/cutover tests should be carried into the private operations repository if that move is still planned. I have not inspected private implementation or asserted that the database migration is complete. This is an ownership/scope closure, not an algorithm correctness review: I did not exhaustively audit the copy/cutover internals or rerun the operational tests.

The separate #29405 changes the active public relay runtime's database-selection contract and is left open; private operator ownership does not invalidate that runtime feature. Relevant public ownership and runtime boundaries are unchanged in HEAD..origin/main.