Summary
drainOrbRelay returns HTTP 500 repeatedly — 872 events, escalating, still firing as of filing.
Evidence (Sentry)
GITTENSORY-1C — Error: orb_relay_drain_http_500, first seen 2026-07-06, last seen 2026-07-11 (today). Culprit drainOrbRelay(server), src/orb/broker-client.ts:255. Tagged jobType: orb-relay-drain, subsystem: scheduled, pendingAckCount: 1, running on a Sentry monitor (gittensory-selfhost-selfhost-orb-relay-drain).
Stack trace shows this is invoked from src/server.ts:1103 (drainRelay) via src/selfhost/monitored-work.ts:75 (drainOrbRelayWithMonitor → withSentryMonitor), i.e. a scheduled/cron-driven drain, not a one-off webhook.
Root cause (Sentry Seer, needs independent verification)
Seer's summary: "Add an in-flight guard to prevent concurrent drain calls, and increase the HTTP timeout beyond the poll interval." Plausible given the scheduled/monitored nature of the call (a slow drain overlapping with the next scheduled tick would race against itself), but verify against the actual poll-interval config and whether a guard already exists (this repo has a claimPrActuationLock-style pattern used elsewhere for exactly this class of problem — check whether the same pattern applies here or is overkill for a single-drain-per-installation cadence).
Requirements
- Confirm whether concurrent
drainOrbRelay calls can actually overlap given the current scheduling, and whether that's the real cause of the 500s (vs. a genuine server-side broker issue — check the broker's own logs/metrics if accessible).
- If it's a race: add an in-flight guard scoped appropriately (likely per-installation).
- If it's a timeout/poll-interval mismatch: fix the relationship so the HTTP call always completes (or times out cleanly) before the next scheduled tick.
- Don't mask a genuine broker-side outage by suppressing the error — the fix should stop the self-inflicted race, not hide a real upstream problem.
Deliverables
Summary
drainOrbRelayreturns HTTP 500 repeatedly — 872 events, escalating, still firing as of filing.Evidence (Sentry)
GITTENSORY-1C —
Error: orb_relay_drain_http_500, first seen 2026-07-06, last seen 2026-07-11 (today). CulpritdrainOrbRelay(server),src/orb/broker-client.ts:255. TaggedjobType: orb-relay-drain,subsystem: scheduled,pendingAckCount: 1, running on a Sentry monitor (gittensory-selfhost-selfhost-orb-relay-drain).Stack trace shows this is invoked from
src/server.ts:1103(drainRelay) viasrc/selfhost/monitored-work.ts:75(drainOrbRelayWithMonitor→withSentryMonitor), i.e. a scheduled/cron-driven drain, not a one-off webhook.Root cause (Sentry Seer, needs independent verification)
Seer's summary: "Add an in-flight guard to prevent concurrent drain calls, and increase the HTTP timeout beyond the poll interval." Plausible given the scheduled/monitored nature of the call (a slow drain overlapping with the next scheduled tick would race against itself), but verify against the actual poll-interval config and whether a guard already exists (this repo has a
claimPrActuationLock-style pattern used elsewhere for exactly this class of problem — check whether the same pattern applies here or is overkill for a single-drain-per-installation cadence).Requirements
drainOrbRelaycalls can actually overlap given the current scheduling, and whether that's the real cause of the 500s (vs. a genuine server-side broker issue — check the broker's own logs/metrics if accessible).Deliverables