+ Two independent signals cover this:{" "}
+ gittensory_github_rest_rate_limit_responses_total counts actual 403/429
+ responses from GitHub, and the{" "}
+ gittensory_jobs_rate_limit_admission_deferred_total /{" "}
+ gittensory_jobs_rate_limit_budget_deferred_total /{" "}
+ gittensory_jobs_rate_limited_by_type_total counters track jobs the queue itself
+ held back before making a request, to avoid tripping a limit. All three job-side
+ counters carry the same three labels — kind (webhook or{" "}
+ background), key_scope (installation,{" "}
+ public, global, or other), and job_type{" "}
+ (the queue job's type, e.g. agent-regate-pr) — so you can break a spike down to
+ exactly which token pool and which job type is under pressure.
+
+ A short burst of deferrals is expected and self-resolving: the queue is deliberately trading
+ a few seconds of delay to avoid a real 429. Treat it as a real problem only once it's
+ sustained — which is exactly what{" "}
+ GittensoryGitHubRateLimitResponses (real 403/429s observed) and{" "}
+ GittensoryQueueRateLimitDeferralsHigh (a sustained deferral rate, not a blip)
+ are tuned to alert on, rather than firing on every brief admission hold.
+
+ If a single key_scope=installation pool is consistently the bottleneck, the fix
+ is usually spreading load across more installation tokens (fewer repos per installation) or
+ raising the GitHub App's own rate-limit tier, not code changes here.
+
+ gittensory_github_response_cache_total (REST) and{" "}
+ gittensory_github_graphql_cache_total (GraphQL) both carry a{" "}
+ result label — hit, miss, set,{" "}
+ coalesced, bypassed, or error — and a{" "}
+ class label identifying the endpoint family. A healthy cache should show most
+ traffic as hit for endpoints that are read repeatedly in one review/maintenance
+ pass (PR reads, check-run lookups); a low hit rate on those specific classes, not the
+ overall average, is the useful signal.
+
+ gittensory_qdrant_errors_total carries an op label (
+ upsert, query, or delete) so you can tell whether
+ indexing or retrieval is failing. GittensoryQdrantErrorRateHigh fires on a
+ sustained error ratio, not an isolated blip.
+
QDRANT_URL (e.g. http://qdrant:6333) is reachable from
+ the app container and the qdrant Compose profile is running.
+ QDRANT_API_KEY is set and matches the Qdrant
+ deployment's configuration.
+ gittensory collection (the
+ fixed collection name self-host always uses) was created with a different embedding model
+ than the one currently configured (AI_EMBED_MODEL). Recreating it — delete
+ the collection and let the next index run recreate it at the current width — is the fix,
+ but it temporarily removes ALL indexed RAG context for every repo until re-indexing
+ completes, so treat it as a deliberate, disruptive step, not a routine one.
+
+ For brokered self-host deployments, gittensory_orb_events_exported_total and{" "}
+ gittensory_orb_export_errors_total track the hourly outcome-export loop;{" "}
+ GittensoryOrbExportErrorRateHigh fires on a sustained error ratio there. The
+ pull-mode relay loop (for installations receiving events outbound from Orb) reports through{" "}
+ gittensory_orb_relay_drains_total (result=events when it drained
+ something, result=empty otherwise) and{" "}
+ gittensory_orb_webhook_total (event + result labels)
+ for what happened to each relayed event once enqueued locally.
+
+ If exports are failing but the relay itself looks healthy, the export loop's Sentry + cron monitor (see Self-host operations) is + the fastest way to confirm whether the loop is even running, before digging into the error + counters. +
+ +
+ Each AI provider (self-host AI_PROVIDER entries) has its own circuit breaker:
+ after 3 consecutive failures it stops attempting real calls to that provider for 60 seconds,
+ recorded as gittensory_ai_provider_circuit_open_total{'{provider="..."}'}{" "}
+ (skipped calls) alongside{" "}
+ gittensory_ai_provider_failures_total{'{provider="..."}'} (real failures). It
+ self-heals automatically — there is no manual reset — but it will reopen immediately if the
+ underlying problem is still there.
+
circuit_open: provider "..." to confirm which provider
+ tripped, and selfhost_ai_provider_failed_in_chain for the real error each
+ failed attempt hit before the breaker opened.
+ selfhost_ai_cli_missing at boot), or the endpoint being genuinely
+ unreachable from the container.
+ GittensoryAiProviderCircuitOpen fires on any circuit-open event in a
+ 15-minute window — a single trip during a real but brief outage is expected; a rule that
+ keeps firing across multiple windows points at the persistent case above.
+ The trace path is app or smoke process → OTEL collector → Tempo → Grafana. Tempo is only diff --git a/test/unit/docs-selfhost-troubleshooting-metric-names.test.ts b/test/unit/docs-selfhost-troubleshooting-metric-names.test.ts new file mode 100644 index 0000000000..15ec7034e5 --- /dev/null +++ b/test/unit/docs-selfhost-troubleshooting-metric-names.test.ts @@ -0,0 +1,42 @@ +import { readFileSync } from "node:fs"; +import { describe, expect, it } from "vitest"; + +// Drift guard (#1943 gate review finding): the self-hosting troubleshooting runbooks reference exact +// Prometheus metric names and alert names. If a metric is ever renamed/removed in src/, or an alert is +// renamed/removed in prometheus/rules/alerts.yml, this test fails instead of the docs silently going stale +// — mirrors the same source-of-truth-diff approach as scripts/check-openapi-settings-parity.mjs (#2556). + +const DOC_PATH = "apps/gittensory-ui/src/routes/docs.self-hosting-troubleshooting.tsx"; +const doc = readFileSync(DOC_PATH, "utf8"); + +// The exact source files that emit every gittensory_*_total metric referenced in the runbooks, per an +// audit against the real incr()/gauge()/observe() call sites (src/selfhost/metrics.ts's API). +const METRIC_SOURCE_FILES = [ + "src/github/client.ts", + "src/github/graphql-cache.ts", + "src/selfhost/queue-common.ts", + "src/selfhost/sqlite-queue.ts", + "src/selfhost/pg-queue.ts", + "src/selfhost/qdrant-vectorize.ts", + "src/selfhost/orb-collector.ts", + "src/selfhost/monitored-work.ts", + "src/selfhost/ai.ts", +]; +const metricSource = METRIC_SOURCE_FILES.map((path) => readFileSync(path, "utf8")).join("\n"); +const alertsSource = readFileSync("prometheus/rules/alerts.yml", "utf8"); + +describe("self-hosting-troubleshooting doc: metric/alert names match source (#1943)", () => { + it("every gittensory_..._total metric name referenced in the doc is actually emitted by the code", () => { + const names = [...new Set([...doc.matchAll(/gittensory_[a-z0-9_]+_total/g)].map((m) => m[0]))]; + expect(names.length).toBeGreaterThan(5); // sanity: the extraction found the runbooks' real content + const missing = names.filter((name) => !metricSource.includes(name)); + expect(missing).toEqual([]); + }); + + it("every GittensoryXxx alert name referenced in the doc exists in prometheus/rules/alerts.yml", () => { + const names = [...new Set([...doc.matchAll(/Gittensory[A-Za-z]+/g)].map((m) => m[0]))]; + expect(names.length).toBeGreaterThan(2); + const missing = names.filter((name) => !alertsSource.includes(`alert: ${name}`)); + expect(missing).toEqual([]); + }); +});