Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions apps/loopover-ui/content/docs/ams-kill-switch-incident.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,13 @@ Flag a misbehaving loop from any of:
3. Fleet observability — error-rate spikes, abnormal claim/submission patterns, elevated
`consecutive_failures` / rejection-reason clustering, or an explicit customer/support escalation
(see the rented-loop escalation path from #4806).
4. **Automated PagerDuty page** (#7666) — when `LOOPOVER_ENABLE_PAGERDUTY` is on and
`PAGERDUTY_ROUTING_KEY` is configured in the **miner** process, a kill-switch **trip** (engage)
fires an Events API v2 page using the same flag / routing-key / enqueue contract as ORB's
`notify-pagerduty` module. A resume does not page. The miner has no D1-backed severity floor or
cooldown (PagerDuty's own `dedup_key` still coalesces duplicate incidents). Treat an
`ams_kill_switch:*` incident as "operator already needed on the page" and start the 15-minute
response clock immediately.

**Triage before acting:** decide one-repo vs fleet-wide. Prefer the narrowest scope that stops the
harm so one bad tenant does not force an unnecessary global halt.
Expand Down
35 changes: 33 additions & 2 deletions packages/loopover-miner/lib/governor-kill-switch.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,42 @@ export type RecordMinerKillSwitchTransitionInput = {
previousScope: MinerKillSwitchScope;
scope: MinerKillSwitchScope;
};
/** Pure page payload for a kill-switch TRIP (#7666). Null unless the transition is an engage into an active scope. */
export type MinerKillSwitchPagerDutyAlert = {
repoFullName: string;
summary: string;
severity: "critical";
dedupKey: string;
customDetails: {
previousScope: MinerKillSwitchScope;
scope: MinerKillSwitchScope;
reason: string;
};
};
/**
* Build the PagerDuty alert for a kill-switch trip. Returns null on resume / same-scope so clearing a halt
* never wakes anyone. `repoFullName` falls back to `ams/fleet` for a global halt with no single-repo context.
*/
export declare function buildMinerKillSwitchPagerDutyAlert(input: {
repoFullName?: string | null | undefined;
previousScope: MinerKillSwitchScope;
scope: MinerKillSwitchScope;
}): MinerKillSwitchPagerDutyAlert | null;
export type NotifyMinerKillSwitchTrip = (alert: MinerKillSwitchPagerDutyAlert, env: Record<string, string | undefined>) => void | Promise<void>;
/**
* Miner-side mirror of `triggerPagerDutyIncident` (#7666): same flag, same global routing key, same Events
* API v2 enqueue. No D1 audit/cooldown (miner has no Worker Env) -- PagerDuty's own `dedup_key` still
* coalesces duplicate incidents. Best-effort: never throws.
*/
export declare function notifyMinerKillSwitchPagerDuty(alert: MinerKillSwitchPagerDutyAlert, env?: Record<string, string | undefined>): Promise<void>;
/**
* Record a kill-switch state transition to the governor ledger. No-op (returns null, appends nothing) when the
* scope has not actually changed since the previous check — callers own tracking the previous scope (in-memory
* or persisted); this module holds no state of its own.
* scope has not actually changed since the previous check -- callers own tracking the previous scope (in-memory
* or persisted); this module holds no state of its own. On a trip, also fires the PagerDuty page (#7666)
* unless `notify` is overridden (tests) or the integration flag/key is unset.
*/
export declare function recordMinerKillSwitchTransition(input: RecordMinerKillSwitchTransitionInput, options?: {
append?: (event: AppendGovernorEventInput) => GovernorLedgerEntry;
notify?: NotifyMinerKillSwitchTrip;
env?: Record<string, string | undefined>;
}): GovernorLedgerEntry | null;
110 changes: 106 additions & 4 deletions packages/loopover-miner/lib/governor-kill-switch.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading