Repository navigation
fix(cloud): recalibrate the relay monitor's postgres-retry freeze to a measured bar - #18580
Conversation
…a measured bar The global relay_cells FOR UPDATE lock made successful retries a steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504 per five minutes over the last 24 h, 55% of windows over the 300 bar, only 22% of 15-minute gates clean. Three read-only dry-runs on 2026-09-04 froze on it, blocking the same-cap roll that carries #18521 and the beginProof crash guard to the 23 cells. 2000 clears every measured healthy gate; the exhausted-retry, director concurrency, and pool bars keep the incident discriminator role.
|
This run was cancelled 🛑 The workflow was cancelled before completion. Please check the link below for details. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (3)
Included review availability: Your plan provides up to 10 included reviews per hour; 3 remain after this review. 📝 WalkthroughWalkthroughThe relay PostgreSQL retry freeze threshold increases from 300 to 2,000 retries per five minutes. Incident-monitor comments describe retries above the threshold as a guard against unbounded growth. Tests verify green status at 1,504 retries and freeze status at 2,001 retries. The incident-monitor documentation updates the threshold table and records the recalibration data and rationale. Merge Risk: ⚪ Minimal · up to The relay monitor now permits up to 2,000 PostgreSQL retries per five minutes before freezing, avoiding freezes during measured healthy activity while retaining the higher retry boundary and other incident checks. The change is ready to merge. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description explains the change, rationale, measured data, scope, and follow-up. However, it omits several required template sections, including an explicit Linked Issue entry in the required format, Visual Proof or N/A, Testing, AI Disclosure, Review, Agent skill upstream boundary, Notes, and Checklist. Resolution Add all required template sections. Include the linked issue as
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…a measured bar (stablyai#18580) The global relay_cells FOR UPDATE lock made successful retries a steady-state rate: fleet-wide p50 430 / p90 924 / p99 1320 / max 1504 per five minutes over the last 24 h, 55% of windows over the 300 bar, only 22% of 15-minute gates clean. Three read-only dry-runs on 2026-09-04 froze on it, blocking the same-cap roll that carries stablyai#18521 and the beginProof crash guard to the 23 cells. 2000 clears every measured healthy gate; the exhausted-retry, director concurrency, and pool bars keep the incident discriminator role.
ELI5
Before any production cell roll, a 15-minute health check has to pass. One rule says "no more than 300 database lock retries per five minutes". The relay's shared lock is now busy enough that the healthy fleet does 400–1,500 retries per five minutes all day, so the check fails and nobody can roll the cells. The cell roll is the thing that makes the lock less busy (and stops a crash that drops ~800 desktops at a time). This PR sets the rule to a number the healthy fleet stays under, while the other rules keep catching real incidents.
What Changed
cloud/apps/relay-ops/src/incident-monitor.ts:relayPostgresRetries300 → 2000 per five minutes, with the measurement basis in the comment.cloud/apps/relay-ops/src/incident-monitor.test.ts: the "301 freezes with threshold 300" test becomes "1504 is green, 2001 freezes with threshold 2000". Reverting the constant to 300 fails it (checked).cloud/docs/relay-incident-monitor.md: threshold table row and an implementation-log entry with the basis.Only the monitor is affected. Terraform alerts do not use this signal (
relay-observability.tfalerts on exhausted retries and pool pressure, not onpostgres_retries).Why
Follow-up to #18569, which recalibrated the exhausted-retry bar. The very next dry-runs froze on this one instead:
relay.postgres_retries380director.concurrency76.7 (six-cell crash storm, real)relay.postgres_retries339, quiet windowMeasured from the
orca_relay_postgres_retrieslog metric (director + cells, summed per five minutes):15-minute gate pass rate over the last 24 h by candidate bar: 300 → 22%, 1000 → 86%, 1500 → 99%, 2000 → 100%.
The 2026-08-23 lock incident on this same metric peaked at 1,510 in one window and 646 in the next, so retries alone no longer separate an incident from today's baseline. That role is carried by
relayPostgresRetryExhausted(incident peak 467 vs bar 300, healthy max 184 in the last 72 h),directorConcurrency(caught the 04:46Z crash storm), and the pool bars.Follow-up
Re-tighten after the same-cap roll puts the fleet on the 500 ms lock wait and the baseline is re-measured. Removing the global lock from per-connection paths is a separate PR.