Skip to content

feat(governor): page PagerDuty on kill-switch trips - #8052

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
galuis116:feat/ams-kill-switch-pagerduty-7666
Jul 22, 2026
Merged

feat(governor): page PagerDuty on kill-switch trips#8052
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
galuis116:feat/ams-kill-switch-pagerduty-7666

Conversation

@galuis116

@galuis116 galuis116 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • On an AMS miner kill-switch TRIP (not a resume), recordMinerKillSwitchTransition (packages/loopover-miner/lib/governor-kill-switch.ts) now also fires a PagerDuty page — fire-and-forget, after the governor-ledger row lands, so a paging failure can never block or mask the ledger write.
  • packages/loopover-engine/src/governor/kill-switch.ts gains a new pure builder, buildMinerKillSwitchPagerDutyAlert, mirroring the existing buildMinerKillSwitchTransitionGovernorLedgerEvent's "no-op unless the scope actually changed" gate, but narrower: it also returns null on a transition into "none" (a resume) — only a trip pages.
  • The miner-side notify (notifyMinerKillSwitchPagerDuty) mirrors src/services/notify-pagerduty.ts's Events API v2 contract exactly (LOOPOVER_ENABLE_PAGERDUTY flag, PAGERDUTY_ROUTING_KEY, dedup_key, enqueue URL/payload shape), with the same no-D1/Worker-Env simplification control-plane/src/pagerduty-notify.ts (Wire real PagerDuty alerting into control-plane provisioning failures #7667) already used for the same reason: the miner is a plain Node process with no Cloudflare Worker Env/D1 binding, so there's no per-repo routing-key map or severity-threshold/cooldown DB query here — PagerDuty's own dedup_key still coalesces duplicate incidents. This is the same pattern the issue asked for reused, not a second alerting mechanism, and not a live call into the hosted module (which can't be imported from a plain Node package).
  • Updated the kill-switch incident runbook's Detection section to note the trip now pages automatically, closing the gap where it assumed a human was "already on the page."

Scope

Validation

  • git diff --check
  • New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries
  • npm run actionlint
  • npm run docs:drift-check
  • npm --workspace @loopover/engine run build (typecheck) + npm --workspace @loopover/engine run test (598/598 passing)
  • npm run build:miner (typecheck for @loopover/miner, includes check-syntax)
  • npx tsc --noEmit -p tsconfig.json (root typecheck)
  • npm --workspace @loopover/ui run typecheck
  • npx vitest run test/unit/governor-kill-switch-pagerduty.test.ts test/unit/miner-governor-kill-switch.test.ts test/unit/kill-switch-incident-runbook.test.ts --coverage — both changed files (packages/loopover-engine/src/governor/kill-switch.ts, packages/loopover-miner/lib/governor-kill-switch.ts) at 100% lines/branches/functions/statements
  • npx vitest run test/unit/miner-attempt-cli.test.ts test/unit/miner-loop-cli.test.ts (existing call sites into the kill-switch module, unaffected — 126/126 passing)

If any required check was skipped, explain why:

  • The whole-repo npm run test:coverage (unsharded) timed out in this sandbox rather than passing or failing — a known sandbox resource constraint, not a signal about this change. Coverage for both changed files was verified directly (100% lines/branches/functions/statements) via a scoped vitest --coverage run against the exact tests that exercise them, per this session's documented workaround for scoped verification.

Safety

  • No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
  • Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
  • Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
  • API/OpenAPI/MCP behavior is updated and tested where needed.
  • UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
  • Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails. SVG screenshots are not used as review evidence. Review-only screenshots or recordings are not committed to the repository.
  • Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

UI Evidence

This is a backend-only fix with no rendered UI delta — before and after are the same production capture at each required viewport. apps/loopover-ui is dark-mode-only, so only the Dark row per viewport applies.

Viewport · Theme Before After
Desktop · Dark Desktop · Dark before Desktop · Dark after
Tablet · Dark Tablet · Dark before Tablet · Dark after
Mobile · Dark Mobile · Dark before Mobile · Dark after

Notes

Wire the miner AMS kill-switch trip path into the existing PagerDuty
alerting pattern instead of a new mechanism. packages/loopover-engine's
kill-switch.ts gains a pure buildMinerKillSwitchPagerDutyAlert builder
(mirrors buildMinerKillSwitchTransitionGovernorLedgerEvent's own
no-op-unless-changed gate, but only on a TRIP, never a resume).
packages/loopover-miner's governor-kill-switch.ts gains the IO wrapper
notifyMinerKillSwitchPagerDuty, mirroring src/services/notify-pagerduty.ts's
Events API v2 contract (LOOPOVER_ENABLE_PAGERDUTY flag, PAGERDUTY_ROUTING_KEY,
dedup_key) with the same no-D1/Worker-Env simplification control-plane's
own mirror (JSONbored#7667) used. recordMinerKillSwitchTransition now pages
fire-and-forget after the ledger row lands, wrapped so a paging failure
can never block or mask the ledger write.

Closes JSONbored#7666
@galuis116
galuis116 requested a review from JSONbored as a code owner July 22, 2026 15:56
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 22, 2026
@loopover-orb

loopover-orb Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-22 16:12:12 UTC

6 files · 1 AI reviewer · no blockers · readiness 95/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR adds PagerDuty paging on kill-switch TRIPs (not resumes), following the established pattern from #7667's control-plane mirror: a pure builder (buildMinerKillSwitchPagerDutyAlert) gates on scope-change and active-scope, and the IO wrapper (notifyMinerKillSwitchPagerDuty) fires fire-and-forget after the ledger write, wrapped in both sync try/catch and async .catch so it can never block or mask the ledger append. The implementation is correct and well-tested — tests verify ledger-before-notify ordering, resume suppression, sync-throw and async-rejection swallowing, and the notify/env injection defaults. It closes the linked issue #7666 and updates the incident runbook accordingly.

Nits — 7 non-blocking
  • The magic numbers (200-char message truncation, 1024-char summary truncation, 5000ms timeout) mirror the existing control-plane/notify-pagerduty.ts conventions but aren't extracted as named constants — minor, matches existing style elsewhere.
  • console.warn calls in governor-kill-switch.ts:42/78 are intentional structured logging for a best-effort failure path, not debug leftovers — no action needed.
  • The hardcoded PAGERDUTY_EVENTS_URL matches the same hardcoding already present in the ORB and control-plane mirrors this PR explicitly follows, so it's consistent rather than a new smell.
  • Consider whether a future PR should extract the shared Events API v2 payload-building logic (routing key regex, truthy-env regex, envString helper) into `@​loopover/engine` to avoid triplicated logic across ORB/control-plane/miner, though the doc comments make clear this triplication is deliberate given each package's differing IO capabilities.
  • The runbook update in ams-kill-switch-incident.mdx is a nice touch closing the gap where detection assumed a human was already watching.
  • Possible screenshot-table issue: identical images (row 1) — Advisory only — verify the screenshot-table images against the stated change before deciding.
  • Possible screenshot-table issue: identical images (row 2) — Advisory only — verify the screenshot-table images against the stated change before deciding.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #7666
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 1931 registered-repo PR(s), 1262 merged, 55 issue(s).
Contributor context ✅ Confirmed Gittensor contributor galuis116; Gittensor profile; 1931 PR(s), 55 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Partially addressed
The PR does add real PagerDuty paging on kill-switch trips, but it explicitly builds a second, parallel PagerDuty integration in the miner package rather than calling src/services/notify-pagerduty.ts's existing integration as the issue explicitly requires ('do not build a second alerting mechanism ... a PR that builds a new alerting path does not satisfy this issue').

Review context
  • Author: galuis116
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: JavaScript, Python, Dart, TypeScript, HTML, MDX, Rust, C++
  • Official Gittensor activity: 1931 PR(s), 55 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Start here: Triage stale or unlinked PRs.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask <question> answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

Visual preview
Route Viewport Before (production) After (this PR's preview) Diff
/ desktop before /
before /
after /
after /
/ mobile before / (mobile)
before / (mobile)
after / (mobile)
after / (mobile)

Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy.

Scroll preview
Route Before (production) After (this PR's preview)
/ before / (scroll)
before / (scroll)
after / (scroll)
after / (scroll)

A short scroll-through clip (desktop) — click either thumbnail to open the full animation. Evidence for scroll-linked behavior a single screenshot can't show.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

@loopover-orb
loopover-orb Bot merged commit 8329069 into JSONbored:main Jul 22, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wire real PagerDuty alerting into AMS kill-switch trips (currently ledger-only, no paging)

1 participant