Skip to content

feat(observability): shared severity-threshold resolver for Sentry + PagerDuty - #5704

Merged
JSONbored merged 1 commit into
mainfrom
feat/severity-threshold-resolver-5119
Jul 14, 2026
Merged

feat(observability): shared severity-threshold resolver for Sentry + PagerDuty#5704
JSONbored merged 1 commit into
mainfrom
feat/severity-threshold-resolver-5119

Conversation

@JSONbored

Copy link
Copy Markdown
Owner

Summary

  • Every self-host observability capture point fired unconditionally before this: captureError/captureReviewFailure always forwarded to Sentry, and forwardStructuredLogToSentry hardcoded an error/fatal-only cutoff with no way to configure it. The one severity-threshold precedent (PagerDuty's PAGERDUTY_MIN_SEVERITY/PAGERDUTY_REPO_MIN_SEVERITY) was scoped to paging only.
  • Extracts a shared resolveSeverityThreshold (src/services/severity-threshold.ts): global env var + per-repo JSON-map override, the same precedence every other alerting channel (Discord, PagerDuty) already uses.
  • Wires it into all three sentry.ts capture paths via new SENTRY_MIN_SEVERITY/SENTRY_REPO_MIN_SEVERITY vars, and retrofits notify-pagerduty.ts's own resolver to delegate to the shared one (kept PagerDutySeverity as a type alias so ops-wire.ts's existing import is untouched).
  • Default threshold (error) is byte-identical to every capture path's pre-existing behavior — an operator who never touches these vars sees no change. Lowering a specific repo's threshold (e.g. to info) surfaces routine structured logs from that repo while it's under active investigation, without raising Sentry noise everywhere else.
  • Regenerated the self-host env reference (npm run selfhost:env-reference) for the 2 new vars.

Test plan

  • test/unit/severity-threshold.test.ts (new) — the shared resolver: repo-map precedence, case-insensitivity, malformed-map handling, empty-repo fallback, process.env fallback
  • test/unit/selfhost-sentry.test.ts — 13 new regression tests: default-unchanged behavior for all 3 capture paths, per-repo suppression/isolation, global-threshold fallback for non-repo-scoped captures, lowering a repo's threshold surfaces warning/info logs, an unrecognized level (e.g. "audit") never gets promoted to error grade
  • test/unit/notify-pagerduty.test.ts + test/unit/ops-wire.test.ts — unaffected, still green (confirms the PagerDutySeverity alias didn't break anything)
  • Full local gate (npm run test:ci + npm audit --audit-level=moderate) green: 831 test files passed, 0 failed

Closes #5119

…PagerDuty

Every self-host observability capture point fired unconditionally
before this: captureError/captureReviewFailure always forwarded to
Sentry, and forwardStructuredLogToSentry hardcoded an error/fatal-only
cutoff with no way to configure it. The one severity-threshold
precedent (PagerDuty's PAGERDUTY_MIN_SEVERITY/PAGERDUTY_REPO_MIN_SEVERITY)
was scoped to paging only, not general Sentry/log verbosity.

Extracts a shared resolveSeverityThreshold (src/services/severity-threshold.ts):
global env var + per-repo JSON-map override, the same precedence every
other alerting channel (Discord, PagerDuty) already uses. Wires it
into all three sentry.ts capture paths (SENTRY_MIN_SEVERITY /
SENTRY_REPO_MIN_SEVERITY) and retrofits notify-pagerduty.ts's own
resolver to delegate to the shared one instead of keeping a parallel
copy. Default threshold (error) is byte-identical to every path's
pre-existing behavior; an operator can lower a specific repo's
threshold to warning/info for full visibility while actively
debugging it, without raising Sentry noise everywhere else.

Regenerated the self-host env reference for the 2 new vars.

Closes #5119
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Preview URL Updated (UTC)
✅ Deployment successful!
View logs
loopover-ui e73dea7 Commit Preview URL

Branch Preview URL
Jul 14 2026, 06:01 AM

@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 95.05%. Comparing base (7515471) to head (e73dea7).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/selfhost/sentry.ts 93.75% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #5704   +/-   ##
=======================================
  Coverage   95.05%   95.05%           
=======================================
  Files         580      581    +1     
  Lines       46127    46157   +30     
  Branches    14793    14808   +15     
=======================================
+ Hits        43845    43874   +29     
  Misses       1523     1523           
- Partials      759      760    +1     
Flag Coverage Δ
shard-1 43.96% <85.00%> (+0.02%) ⬆️
shard-2 35.62% <2.50%> (+<0.01%) ⬆️
shard-3 32.28% <2.50%> (-0.03%) ⬇️
shard-4 32.84% <52.50%> (-0.18%) ⬇️
shard-5 31.35% <55.00%> (-0.23%) ⬇️
shard-6 44.68% <30.00%> (+0.18%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/services/notify-pagerduty.ts 100.00% <100.00%> (ø)
src/services/severity-threshold.ts 100.00% <100.00%> (ø)
src/selfhost/sentry.ts 99.31% <93.75%> (-0.33%) ⬇️
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JSONbored
JSONbored merged commit 24800c6 into main Jul 14, 2026
19 of 20 checks passed
@JSONbored
JSONbored deleted the feat/severity-threshold-resolver-5119 branch July 14, 2026 06:05
@loopover-orb loopover-orb Bot added the gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier. label Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:bug Gittensor-scored bug fix — scores a 0.05x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Global + per-repo alerting/log severity thresholds for the ORB stack

1 participant