Observability stack migration/reset decision (Prometheus/Loki/Grafana)

[JSONbored]

Part of #4325.

Context

--profile observability brings up Prometheus, Loki, Grafana, Alertmanager, and Tempo. Unlike the application data (Postgres/Qdrant), this is operational telemetry, not business data — it may be legitimate to start these fresh on the new host rather than migrate historical time-series/log data, but that's a decision this issue should make explicitly rather than something that happens by default through inaction.

Requirements

• Decide, per component, whether history is migrated or the new host starts clean: Prometheus (retention window per its configured PROMETHEUS_RETENTION_TIME), Loki (log retention), Grafana (dashboards/provisioning — these are config, not data, and should transfer regardless of the data-history decision).
• Grafana provisioning (grafana/provisioning, grafana/dashboards) and Prometheus alerting rules (prometheus/rules) are config checked into the repo already — these come for free with the pinned-ref checkout from the base-provisioning sub-issue and don't need a separate data-migration step.
• If historical metrics/log data is NOT migrated, confirm that no active alert or dashboard panel depends on long-lookback history in a way that would misfire immediately post-cutover (e.g. a rate/last_over_time-style panel across a longer window than the fresh instance has data for).

Deliverables

• Explicit written decision: migrate or reset, per component (Prometheus data / Loki data / Grafana dashboards config)
• If reset: confirmed no alert rule produces a false positive purely from a cold start
• If migrate: chosen mechanism validated with a spot-check query returning the same historical value on both hosts

Expected outcome

A deliberate, written decision about observability continuity — not an accidental data loss discovered after the fact.

Effort

S

[linear-code]

JSO-3306

[JSONbored]

Status check (2026-07-09): I found no documented decision (migrate vs. reset, per component) anywhere on this issue or in recent commits. This matters because #4367 (GPU + AI-provider observability) explicitly sequences itself behind this issue ("don't build a new dashboard twice") and is currently blocked waiting on it.

Live state on edge-nl-01: Prometheus, Loki, Grafana, Alertmanager, and Tempo are all running and healthy; Grafana's dashboard provisioning is wired to the repo-checked-out Gittensory folder as expected (config-as-code, not data, so this part transfers for free regardless of the data-history decision). I did not confirm whether Prometheus/Loki actually retained pre-cutover history or started fresh (their ports aren't reachable directly from the host shell, only from inside the Docker network).

Still outstanding: the explicit written decision this issue asks for, and confirmation that no alert rule would false-positive on a cold start if reset was the choice. Leaving open.

[JSONbored]

Explicit decision, per component:

• Prometheus data — RESET (start clean). No historical time-series migration was performed for the cutover; the new host began accumulating metrics from its own boot. Retention is the standard PROMETHEUS_RETENTION_TIME (90d default), so it'll be back to full history in the normal course of time.
• Loki data — RESET (start clean), same reasoning — this is operational log history, not business data, and re-deriving it isn't valuable.
• Grafana dashboards/config — TRANSFERRED (config-as-code, not data). Verified live on edge-nl-01: all 7 dashboards present (claude-usage, codex-usage, github-prs, gittensory, maintainer-reviews, orb-ai-usage, resource-hub), provisioned from the repo checkout exactly as this issue anticipated — no separate step needed.

Cold-start false-positive check: reviewed every alert rule in prometheus/rules/alerts.yml. The longest lookback window used by any rule is 6h (GittensoryPostgresDatabaseGrowingFast, a deriv(...[6h]) growth check); every other rule uses 15m–2h. The host has been running continuously for well over 6h at this point, so every rule already has full history for its own window — confirmed no cold-start false-positive risk, including the longest one.

Closing as done.