[Jinwoo-H]

	Files	Added	Deleted	Net
Test	2	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​290	$\color{#cf222e}{\Huge{\mathbf{−}}}$​9	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​281
Prod	4	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​190	$\color{#cf222e}{\Huge{\mathbf{−}}}$​21	$\color{#1a7f37}{\Huge{\mathbf{+}}}$​169

ELI5

Before we change anything in the relay fleet, a 15-minute safety check runs and
looks at a long list of production health signals. If any of them look wrong, it
stops the change.

That check has been stopping us for the wrong reason. It has run 39 times and
stopped 39 times, every single time on a problem that was already there before
the change and has nothing to do with it. So it has not been protecting anything
for the last 8.5 hours, it has just been saying "no" to everything.

This PR teaches the check to ignore three kinds of everyday background noise it
was mistaking for danger, while still stopping on the real thing.

What Changed

Before: the pre-drain dry-run froze on every run. Of the last 14 freezes, 7
were an Asia cell's health probe reading zero for one sample, 5 were the director
reporting 4-9 server errors against a bar of 3, and 2 were the auth database
holding 272-281 connections against a bar of 250.

After: the same run passes on those conditions and still freezes on the
failures it exists to catch.

Three changes, all in the monitor:

1. A cell health probe now has to fail repeatedly, not once. Each sample is a
single HTTP round trip from one GitHub runner to one cell. The Asia cells run
their readiness check as a SELECT 1 against Cloud SQL in us-central1, a 176 ms
round trip, behind a 2-second timeout. When the connection pool is busy that
check times out, the load balancer answers "no healthy upstream" for about 30
seconds, and the monitor recorded health=0 and ready=0.

The important detail is that "no healthy upstream" arrives as a real HTTP 503
response, not as a dropped connection. So the monitor cannot tell it apart from a
cell that is genuinely serving an unhealthy answer by looking at where the
answer came from. What does tell them apart is how long it lasts: the load
balancer case clears in about 30 seconds, and the sample interval is 60 seconds,
so it shows up in one sample and at worst two. A genuinely broken cell stays
broken.

So per-cell probe signals now need to fail more than cellProbeToleranceSamples
(2) consecutive samples before they freeze the run. The director and auth probes
are deliberately excluded and keep zero tolerance, because they are the single
points of failure this gate mainly exists to catch. A blip that gets absorbed is
written into the sealed state artifact under toleratedProbeEvents, so a passing
run still shows exactly what the gate chose not to stop on.

The streak is keyed by cell, not by signal. A cell's health, ready and latency
readings all describe the same round trip, so they share one streak. Keyed per
signal instead, a cell that alternates between answering slowly and not answering
at all would hold every individual streak at one and never reach the tolerance,
which would let a continuously unhealthy cell through. There is a test for exactly
that shape.

The same tolerance applies to the live preflight. A separate check collects one
sample immediately before every mutating wave, and it previously retried only on
freshness codes. That left the identical hole: the same 30-second blip could pass
the 15-minute gate and then fail the wave seconds later. It now re-samples per-cell
probe breaches up to the same tolerance, spaced by the existing retry interval and
still capped by the same evidence-age budget. Director and auth probes, and every
non-probe threshold, still fail the wave on the first bad sample.

2. directorErrors 3 -> 15. This counts non-503 5xx answers from the
director over a rolling five-minute window.

3. cloudSqlBackends 250 -> 320. This is the connection count on the auth
database instance, summed across its seven databases.

Failure codes in the artifact are unchanged, so the tooling that matches on the
source/code prefix keeps working. The trusted evidence scripts under
cloud/dev/scripts/ are not touched by this PR.

cloud/docs/relay-incident-monitor.md is updated to match: the threshold table,
the stale 400-connection ceiling text, a note on the probe tolerance, and a dated
2026-09-17 entry in the recalibration log alongside the prior ones. The superseded
2026-08-26 backends entry and 2026-09-12 director-error note now say so.

Why

This loosens a safety gate. Here is why it is still safe.

A gate that has refused 39 of 39 runs is not adding safety, it is only adding
delay, and the pressure it creates is to bypass it by hand, which is strictly
worse than a calibrated gate. Each new bar below is set from measured production
data over the 24 hours to 2026-09-17, read through the Cloud Monitoring API with
the same aggregation the monitor itself uses.

director.errors, non-503 5xx per rolling 5 minutes:

p50	p90	p95	p99	max
0	3	5	9	52

The old bar of 3 sat exactly on the p90, so 9.2% of individual windows and 29.0%
of 15-minute gates froze on ordinary load. 15 sits above the chronic p99 and
brings the baseline freeze rate to 1.5%. It is deliberately not set above the
20-52 bursts, so those remain detectable. For scale, the director answers roughly
50 requests a minute in 503s alone, so a director that is actually broken serves
5xx in the hundreds per window, an order of magnitude clear of this bar.

cloud_sql.backends, summed per minute:

p50	p90	p95	p99	max
118	165	212	262	282

The old bar of 250 was below the observed maximum, which is why it could never
be reliably cleared: 1.95% of minutes and 21.8% of 15-minute gates tripped on it.
320 clears every measured healthy minute with 13% of headroom above the peak, and
still fires at 65% of the 490 connections the budget work (#21165 / orca-cloud
#487) treats as usable out of max_connections 500, leaving 170 connections in
hand. Connection exhaustion is a runaway, not a plateau, so it crosses 320 on its
way to 490.

Alternatives considered.

Leave the bars and bypass the gate by hand. Rejected: it throws away the
evidence trail for every future roll, not just this one.

Make the thresholds dispatch-time inputs in the job YAML. Rejected, even though
the brief suggested keeping them configurable where they already are. They are
not currently configurable anywhere: they are code constants, and the whole set
is sealed into every checkpoint artifact as thresholds. That is what makes a
recorded green run auditable after the fact. Moving them to workflow inputs would
let an operator loosen the gate at dispatch time with no review and no stable
record of what bar was actually applied. Changing them in code, with the
measurement written next to the number, is the property worth keeping.

Discriminate the cell probe by provenance instead of persistence, that is, treat
a failed fetch differently from a served health=0. Rejected after reading the
probe: probePath already retries a thrown fetch once and only then reports
ok: false, and the actual production symptom arrives as an HTTP 503 from the
load balancer, which is a real answer. Provenance cannot separate the two cases
here. Persistence can, which is also the discriminator the file already uses for
freshness failures.

Raise the cell latency bar again. Rejected: that was done on 2026-09-05 for this
same pair of cells and did not address health/ready reading zero, which is the
signal actually freezing the gate now.

The design reuses the shape already in this file. INCIDENT_FRESHNESS_TOLERANCE_SAMPLES
already tolerates N consecutive unreadable samples per signal key; the new probe
tolerance is the same per-key streak idea, with the difference that exceeding it
freezes the run rather than restarting the window.

Linked Issue

No GitHub issue. This came from the relay same-cap rollout being blocked for 8.5
hours on the pre-roll gate. cloud/docs/relay-reconnect-2026-09-findings.md
records the same recommendation from an earlier session, where ten dry-runs
produced four freezes on directorErrors alone and recalibration was written up
as follow-up work that was never done.

Visual Proof

N/A. This changes a CI safety gate's arithmetic and has no user-facing surface.

Testing

pnpm --filter @orca-cloud/relay-ops test and pnpm --filter @orca-cloud/relay-ops typecheck, both green at 12 files and 106 tests.

Six new tests in incident-monitor.test.ts drive the probe tolerance end to end
through runIncidentMonitor:

• an outage no longer than the tolerance passes the dry-run, and the absorbed
  blip is still recorded in the artifact
• an outage one sample longer freezes it
• repeated single-sample outages separated by healthy samples never accumulate
• a cell alternating between slow and unanswered freezes, which is the case that
  per-signal keying would have let through
• a resumed state file carrying a full streak freezes on the next bad sample
• the director probe still freezes on a single bad sample

Four new tests in incident-live-preflight-cli.test.ts cover the preflight side:
an outage within the tolerance lets the wave proceed after re-sampling, an outage
past it fails the wave, and neither a director probe failure nor a non-probe
threshold breach causes any re-sampling at all.

The two existing threshold tests were updated to the new bars and now assert the
measured healthy peak passes.

I verified the tests bite rather than passing vacuously, by mutating the source and
confirming the intended tests fail each time:

Mutation	Tests that fail
tolerance set to 0	2
cell-probe classifier forced false	3
cell-probe classifier forced true	1
streak keyed per signal, the reported bug	2
resumed streak ignored	3
preflight cell-probe retry removed	1
preflight retrying every non-freshness failure	7

cloud/dev/scripts/relay-monitor-evidence.test.mjs and
relay-production-capacity-wave.test.mjs also pass at 19 tests, confirming the
state artifact still satisfies the trusted evidence validators.

Not verified: cloud/apps/push fails typecheck on this branch, on
retryLockTimeout not existing in SchemaStartupOptions. That file is untouched
by this PR (git diff origin/main -- cloud/apps/push is empty), so it is
pre-existing on main.

Platforms: this is Node code that runs on the CI Ubuntu runner only. Verified on
macOS locally.

• I manually tested these changes locally
• Automated tests added/updated, or explained why not below

AI Disclosure

Review

Note on evidence authority: editing the monitor's code invalidates any sealed
canary authority that depends on these paths. No canary evidence is sealed at the
time of writing, so nothing is invalidated by this PR. Any dry-run evidence sealed
before this merges was produced under the old bars and should not be reused to
justify a roll after it.

The state file gains two fields, probeStreaks and toleratedProbeEvents. Both
are optional with defaults in the CLI's zod schema, so schemaVersion stays at 4
and a state file written by the previous version still restores. This follows the
tolerated: z.boolean().default(false) precedent already in that schema. Both
evidence validators check fields individually rather than strictly, so the added
fields do not affect them.

Agent skill upstream boundary

• Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

Security: no change to identity, tokens, or admin call paths. The gate is
read-only. Backwards compatibility: covered above, state files restore across the
change. Performance: the streak bookkeeping is a per-sample map operation over at
most a handful of keys. SSH/mobile/cross-platform: not applicable, this runs only
in the production monitor workflow.

Checklist

• This PR is small and focused
• I explained what changed and why (ELI5, the user-facing before/after, the mechanism, and why over the alternatives)
• Before/after screenshots or videos attached for UI changes, or N/A with reason
• Self-reviewed for correctness, security, and performance
• Cross-platform, SSH/remote, and path/shortcut impact considered (or N/A)
• pnpm lint, pnpm typecheck, pnpm test, and pnpm build pass (or CI will cover; local preferred)

[pullfrog]

ℹ️ No critical issues — minor suggestions inline. The threshold recalibration is well-evidenced and the monitor-side per-cell tolerance is sound; the two notes below are about the live preflight not quite matching the monitor's tolerance in practice.

Reviewed changes

• Cloud SQL backends 250 → 320 and director 5xx 3 → 15. Measured 24 h percentiles are recorded next to each constant and in cloud/docs/relay-incident-monitor.md; the new cloudSqlBackends bar also aligns with the pre-existing CLOUD_SQL_LIMIT = 320 in relay-asia-rollout-evidence.mjs. Bounds are exercised exactly (282 green / 321 freeze; 15 green / 16 freeze).
• Per-cell active-probe tolerance in incident-monitor.ts. cellProbeStreakKey keys by cell so health/ready/latency_ms share one streak, director/auth probes return null and stay zero-tolerance, and absorbed breaches are sealed under toleratedProbeEvents. I traced every write to probeStreaks: a cell that fails on every fresh sample it is evaluated against cannot escape freezing on the third.
• Live preflight re-sampling for cell-probe breaches with independent freshness/probe budgets, so a single pre-wave sample no longer hard-fails on a probe blip.
• State compatibility. probeStreaks/toleratedProbeEvents are optional-with-defaults in the CLI StateSchema, schemaVersion stays 4, and the evidence validators check fields individually so the manifest hash chain is unaffected.

️ Nitpicks

• cloud/infra/terraform/relay-observability.tf:622 (out of diff, so not anchored) still documents the alert as firing before the "400-connection ceiling" / "modeled 385-connection operating maximum", while this PR rebases the shared model on max_connections 500 / 490 usable (relay-cloud-sql-connection-budget.test.mjs now expects operatingMaximum 403). Since the PR deliberately refreshed the stale ceiling text in the monitor doc, this sibling reference is now inconsistent with it.

  ^{｜ Fix all ➔ ｜ Fix 👍s ➔ ｜ View workflow run ｜ Using DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏}

[pullfrog]

The preflight copies the monitor's sample count but not its cadence: attempts is 3 here, and retries are spaced by FRESHNESS_RETRY_INTERVAL_MS (15 s), while the monitor needs 3 samples 60 s apart. So the preflight tolerates ~30 s of continuous probe failure versus the monitor's ~120 s, and the cited ~30 s "no healthy upstream" window can still land in three 15 s samples and fail the wave — the outcome this change targets. If parity is the goal, scale the attempt count by the cadence ratio or bound the loop by a wall-clock deadline.

Technical details

# Preflight probe tolerance is ~4x stricter than the monitor's in wall-clock time

## Affected sites
- `cloud/apps/relay-ops/src/incident-live-preflight-cli.ts:178` — `cellProbeAttempts = 1 + cellProbeToleranceSamples` (= 3).
- `cloud/apps/relay-ops/src/incident-live-preflight-cli.ts:20` — `FRESHNESS_RETRY_INTERVAL_MS = 15_000`.
- `cloud/apps/relay-ops/src/incident-monitor.ts:142,904,907,988` — freeze needs `streak > 2` at `state.intervalMs` (60 s), i.e. 3 samples / ~120 s.

## Required outcome
- A ~30 s per-cell probe outage should not fail a mutating wave, matching the monitor gate's stated intent.

## Suggested approach
- Derive the retry count from the cadence ratio (`1 + tolerance * intervalMs / retryIntervalMs`), re-sample probe breaches on the monitor interval, or bound the loop by a wall-clock deadline tied to the tolerated outage length.

[pullfrog]

cellProbeRetries is a single counter shared across all cells, but the monitor's tolerance is a per-cell streak. Two different cells that each blink once on successive samples draw down both retries and throw on the third, even though the monitor would keep two independent streaks of 1 and tolerate both. Since the comment above claims to select per-cell breaches only, keying this budget by cellProbeStreakKey(failure) would match the monitor's behavior.

Technical details

# Preflight probe budget is global, not per cell

## Affected sites
- `cloud/apps/relay-ops/src/incident-live-preflight-cli.ts:181,196-198,215` — one flat `cellProbeFailures` list and one `cellProbeRetries` counter across all cells.
- `cloud/apps/relay-ops/src/incident-monitor.ts:892-905` — monitor streaks are keyed per `cellProbeStreakKey`.

## Required outcome
- Single-sample breaches from different cells must not combine into a tolerance breach for any one cell.

## Suggested approach
- Track a per-key consecutive-breach count in the preflight and throw only when a single key exceeds `cellProbeToleranceSamples`.