Skip to content

Add a native PagerDuty receiver to Alertmanager for severity:critical alerts #5009

Description

@JSONbored

Summary

alertmanager.local (the live, gitignored deploy config on edge-nl-01) already routes every alert to Discord, and has a commented-out generic webhook_configs block labeled "PagerDuty bridge" ready for when PagerDuty is set up. Alertmanager has a first-class, native pagerduty_configs receiver type — use that instead of a generic webhook bridge.

Requirements

  1. Set up a PagerDuty service (Events API v2 integration) for this project, and add a native pagerduty_configs receiver to alertmanager.local using its routing key — not the generic webhook_configs bridge currently stubbed in.
  2. Un-comment and wire the severity-split route already sketched in the file's own comments: severity: critical → PagerDuty (short group_wait, tighter repeat_interval), everything else stays on Discord.
  3. Verify send_resolved behavior end-to-end (PagerDuty auto-resolves when Alertmanager reports the alert cleared) — this is default-on per Alertmanager but worth confirming against the actual PagerDuty integration type chosen (Events API v2 vs. the legacy "Prometheus" v1 integration type use different fields — pick v2 with routing_key, not v1's service_key, since v2 is current).
  4. Keep the backup: .gittensory.yml-style timestamped backup of alertmanager.local before editing, matching this box's existing convention for live config edits.

Deliverables

  • pagerduty_configs receiver live and verified (a test alert actually pages).
  • Severity-based routing confirmed working (critical → PagerDuty, warning → Discord only).
  • Config change on the box, documented (this lives in a gitignored deploy file, not the repo — note where the change actually lives for the next person who looks for it in git and doesn't find it).

Part of #5007.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions