Skip to content

docs(ops): finish self-host rate-limit dashboards and runbooks #1943

Description

@JSONbored

Parent: #1936

Problem

The self-host dashboard is useful but still has blind spots. Beta operators need an operations runbook that explains queue growth, admission deferrals, rate-limit recovery, AI provider failures, Qdrant behavior, and Orb telemetry status.

Requirements

  • Dashboard panels should line up with current metric names and labels.
  • Alerts should distinguish backlog caused by local failure from backlog caused by GitHub admission deferral.
  • Troubleshooting docs should include exact log event names and PromQL examples.
  • Missing/no-data panels should either be wired correctly or documented as optional/profile-dependent.

Deliverables

  • Update Grafana dashboard panels for queue, GitHub API budget, cache, admission deferrals, Orb relay/export, provider failures, Qdrant, and HTTP latency.
  • Update alert rules for sustained backlog, rate-limit response spikes, admission deferral duration, dead-letter growth, and provider outage.
  • Update self-host operations/troubleshooting docs with copy-paste checks.
  • Add validation for dashboard JSON / alert rule syntax if available.

Acceptance criteria

  • An operator can explain a growing queue from Grafana alone.
  • Alerts do not page on expected temporary admission deferrals unless they become sustained/stale.
  • No high-cardinality labels are introduced.
  • Docs match current metric names.

Expected outcome

Beta users can operate the stack without maintainer SSH access, and we can support incidents from screenshots/metrics instead of guesswork.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

Status
Done

Relationships

None yet

Development

No branches or pull requests

Issue actions