feat(ops): add fatigue-controlled PagerDuty paging for ops anomalies

[JSONbored]

Summary

• Adds an experimental, default-OFF PagerDuty integration (src/services/notify-pagerduty.ts) as a direct app-level notifier, mirroring notify-discord.ts's exact per-repo routing precedence: PAGERDUTY_REPO_ROUTING_KEYS (JSON map) over the global PAGERDUTY_ROUTING_KEY fallback, gated by GITTENSORY_ENABLE_PAGERDUTY.
• Hooks the existing ops_anomaly detector in src/review/ops-wire.ts (the codebase's own "something needs a human" signal) rather than paging on every captureError/captureReviewFailure call, which would need its own frequency policy first.
• Alert-fatigue controls, since paging is the loudest channel this codebase has: a global + per-repo minimum severity floor (PAGERDUTY_MIN_SEVERITY/PAGERDUTY_REPO_MIN_SEVERITY, default error, so routine calibration nudges like a gate false-positive spike never page — only active-incident anomalies like a review/failure burst do), and a global + per-repo cooldown window (PAGERDUTY_COOLDOWN_MINUTES/PAGERDUTY_REPO_COOLDOWN_MINUTES, default 60m) between repeat pages for the same ongoing condition. Both are on top of PagerDuty's own dedup_key coalescing, which prevents duplicate incidents but not duplicate pages for a still-open one.
• Pages at the worst (highest-severity) anomaly found in a tick, not whichever anomaly happened to sort first.

Closes #4937.

Scope

• The PR title follows type(scope): short summary Conventional Commit format.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• Linked issue: Closes Wire on-call/paging for the central hosted service #4937 (open, maintainer-authored PR).

Validation

• git diff --check
• npm run actionlint (via npm run test:ci)
• npm run typecheck
• npm run test:coverage locally — both new files (notify-pagerduty.ts, and the changed lines in ops-wire.ts) are 100% line + branch covered on the diff.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate — 0 vulnerabilities.
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries — see test/unit/notify-pagerduty.test.ts (28 tests) and the new runOpsAlerts PagerDuty wiring tests in test/unit/ops-wire.test.ts.

Ran the full npm run test:ci gate locally (all steps green) plus npm audit --audit-level=moderate.

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests. (N/A — no auth/session/CORS surface touched.)
• API/OpenAPI/MCP behavior is updated and tested where needed. (No public API/OpenAPI surface changed — this is an internal ops cron notifier.)
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks. (N/A — no UI changes.)
• No visible UI change — UI Evidence section not applicable.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs. Regenerated apps/gittensory-ui/src/lib/selfhost-env-reference.ts via npm run selfhost:env-reference after adding src/services/notify-pagerduty.ts to the generator's scanned source roots (mirroring the existing notify-discord.ts entry), so the new self-host env vars are documented.

Notes

• No new GitHub issue filed — reuses Wire on-call/paging for the central hosted service #4937 ("Wire on-call/paging for the central hosted service"), which this PR resolves directly. Wire Sentry to real-time notification (Discord/PagerDuty), not the email-only default #5008/Add a native PagerDuty receiver to Alertmanager for severity:critical alerts #5009 describe complementary Sentry-native/Alertmanager-native approaches that remain valid separate follow-ups a human can do via those tools' own web UIs; this PR is the direct app-level approach the user selected instead.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	c28295a	Commit Preview URL Branch Preview URL	Jul 11 2026, 03:39 PM

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.22%. Comparing base (01b37d0) to head (c28295a).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5116      +/-   ##
==========================================
+ Coverage   94.21%   94.22%   +0.01%     
==========================================
  Files         470      471       +1     
  Lines       39743    39821      +78     
  Branches    14506    14533      +27     
==========================================
+ Hits        37442    37520      +78     
  Misses       1645     1645              
  Partials      656      656              

Flag	Coverage Δ
shard-1	46.34% <6.41%> (-0.08%)	⬇️
shard-2	33.83% <19.23%> (-0.03%)	⬇️
shard-3	32.21% <6.41%> (+0.07%)	⬆️
shard-4	31.89% <6.41%> (+0.07%)	⬆️
shard-5	33.58% <88.46%> (+0.05%)	⬆️
shard-6	44.93% <60.25%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/review/ops-wire.ts	98.50% <100.00%> (+0.23%)	⬆️
src/services/notify-pagerduty.ts	100.00% <100.00%> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Caution

🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥🟥

🛑 Gittensory review result - fixes required

_{Review updated: 2026-07-11 15:37:13 UTC}

7 files · 2 AI reviewers · 1 blocker · readiness 100/100 · CI failing · unstable

🛑 Suggested Action - Manual Review

• AI reviewers agree on a likely critical defect: src/services/notify-pagerduty.ts: triggerPagerDutyIncident's cooldown check calls countRecentAuditEventsForActorAndTarget(env, "gittensory", "external_notification.pagerduty", params.dedupKey, cooldownSinceIso) with no outcome filter, so a prior 'denied' audit row (e.g. from a below-min-severity anomaly a few minutes earlier on the same fixed dedupKey `ops_anomaly:${repoFullName}`) counts toward the cooldown and can suppress a genuinely-qualifying later page for the same repo within the cooldown window, even though no page was ever actually sent. — Resolve the flagged defect, or override if the AI reviewers are mistaken, then re-run the gate.

Review summary
Adds an experimental, default-OFF PagerDuty paging integration mirroring notify-discord.ts's routing precedence, hooked into the existing ops_anomaly detector with a min-severity floor and cooldown as fatigue controls. The core mechanics (routing resolution, severity gating, worst-anomaly selection) are correct and well-tested, but the cooldown check appears to count ALL audit rows for a dedupKey (including 'denied' rows from min-severity/invalid-key rejections), not just successfully-sent pages, which can self-suppress a genuine later page. The change is narrowly scoped, flag-gated, and closes #4937.

Blockers

• src/services/notify-pagerduty.ts: triggerPagerDutyIncident's cooldown check calls countRecentAuditEventsForActorAndTarget(env, "gittensory", "external_notification.pagerduty", params.dedupKey, cooldownSinceIso) with no outcome filter, so a prior 'denied' audit row (e.g. from a below-min-severity anomaly a few minutes earlier on the same fixed dedupKey `ops_anomaly:${repoFullName}`) counts toward the cooldown and can suppress a genuinely-qualifying later page for the same repo within the cooldown window, even though no page was ever actually sent.
• src/services/notify-pagerduty.ts:177 counts every recent `external_notification.pagerduty` audit event for the dedup key, then src/services/notify-pagerduty.ts:179 records `cooldown_active` as another matching event, so an ongoing anomaly that is checked every cron tick keeps refreshing its own cooldown and can suppress later repeat pages indefinitely; change the cooldown query to count only prior completed page sends, or use a distinct event type/target for denied cooldown audits.

Nits — 6 non-blocking

• src/services/notify-pagerduty.ts: several new magic numbers (1024-char summary cap, 5000ms fetch timeout, 200/280-char log truncation, 60-minute default cooldown) would read clearer as named constants.
• src/services/notify-pagerduty.ts: triggerPagerDutyIncident nests to depth 5 around the min-severity/cooldown checks; extracting the gating checks into a small helper would flatten this.
• src/review/ops-wire.ts: triggerPagerDutyIncident (a network fetch plus a DB query) is awaited sequentially inside the per-repo anomaly loop, so a tick with several simultaneously-anomalous repos serializes PagerDuty calls (bounded by the 5s per-call timeout) — likely fine given anomalies are rare, but worth a comment if that assumption changes.
• The PR comment in ops-wire.ts already flags that no 'resolve' event is sent once an anomaly clears, leaving incidents to be closed manually — worth a tracking follow-up rather than leaving it implicit.
• Have countRecentAuditEventsForActorAndTarget's caller (or a repository-layer variant) filter to outcome='completed' rows only when used for cooldown, so denied/rejected attempts don't pollute the window — add a regression test seeding a 'denied' audit row moments before a qualifying trigger to confirm the page still fires.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.

Why this is blocked

• src/services/notify-pagerduty.ts: triggerPagerDutyIncident's cooldown check calls countRecentAuditEventsForActorAndTarget(env, "gittensory", "external_notification.pagerduty", params.dedupKey, cooldownSinceIso) with no outcome filter, so a prior 'denied' audit row (e.g. from a below-min-severity anomaly a few minutes earlier on the same fixed dedupKey `ops_anomaly:${repoFullName}`) counts toward the cooldown and can suppress a genuinely-qualifying later page for the same repo within the cooldown window, even though no page was ever actually sent.

CI checks failing

• Workers Builds: gittensory-ui — Workers Builds: gittensory-ui

Signal	Result	Evidence
Code review	❌ 1 blocker	2 reviewers, synthesized
Linked issue	✅ Linked	#4937
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 46 registered-repo PR(s), 38 merged, 416 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 46 PR(s), 416 issue(s).
Gate result	❌ Blocking	Repo-configured hard blocker found.
Improvement	✅ Minor	risk: clean · value: minor — Code changes are accompanied by test evidence. LLM value judgment: moderate — The PR adds a well-scoped, flag-gated, thoroughly-tested paging capability that reuses an existing detector and routing pattern rather than introducing new surface area, directly closing the linked issue.

Linked issue satisfaction

Addressed
The issue asks for a working on-call/paging integration triggered by real incident conditions for the hosted service, and this PR wires PagerDuty's Events API v2 directly into the existing ops_anomaly detector (runOpsAlerts) with severity gating and cooldown controls, hitting real incident data rather than a placeholder.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: not available
• Official Gittensor activity: 46 PR(s), 416 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

[BETA] Chat with Gittensory

Ask Gittensory a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

• @gittensory ask <question> answers contribution-quality Q&A with source citations and freshness.
• @gittensory chat <question> answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
• A plain-language @gittensory mention with a real question is routed to the closest matching read-only command automatically -- no exact syntax required.

Full command reference: https://gittensory.aethereal.dev/docs/gittensory-commands

Visual preview

Route	Viewport	Before (production)	After (this PR's preview)	Diff
/	desktop			—
/	mobile			—

Click any thumbnail to open the full-size screenshot. Before = production · After = this PR's preview deploy.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review