Repository navigation
feat(otel): ClickHouse-first OTEL metrics storage with full-fidelity decode - #158
Merged
Merged
Conversation
…decode Add a native ClickHouse metrics path alongside the existing TimescaleDB one. When TEMPS_CLICKHOUSE_* is configured, OTLP metrics are decoded at full fidelity (temporality, monotonicity, explicit/exponential histograms, summaries, exemplars, typed labels) and stored in a native `metrics` MergeTree; query_metrics/list_metric_names and the anomaly-detector helpers run natively against it. The service_metrics alerting bridge and the default (no-ClickHouse) TimescaleDB path are unchanged. Proven end-to-end against a live ClickHouse testcontainer: store -> query -> read-back round-trips with fidelity (4 CH storage tests + 1 decode->store fidelity test + 263 lib unit tests + 7 e2e + 15 timescale, all green). Correctness fixes uncovered by the live integration tests: - ReplacingMergeTree ORDER BY now includes a MATERIALIZED label fingerprint (sipHash64 of the sorted label set) and a full-precision timestamp. Previously distinct label-series sharing a coarse per-second timestamp silently collapsed into one row (data loss across series). - Bucket expression uses toInt64(toUnixTimestamp(...))*1000; the prior toUnixTimestamp64Milli() over a DateTime was an illegal-type error. - Aggregates wrap value in assumeNotNull so Nullable(Float64) results match the f64 row read (RowBinary type-width mismatch otherwise). Test infrastructure: fix the ClickHouse testcontainer wait strategy (HTTP /ping via the http_wait_plain feature) and credentials so the integration tests actually execute instead of silently skipping. Frontend: scaffold a metrics explorer page (mirrors Traces) wired to the generated SDK; the richer Phase C query params and the Observe metric kind await an SDK regen against a running server. Deferred follow-ups: rate() delta-vs-cumulative handling, histogram quantile reconstruction, live-CH coverage for exp-histogram/summary/exemplar columns, and TimescaleDB parity for the new fidelity.
- rate() now honours aggregation temporality: DELTA series sum their per-interval increments while CUMULATIVE counters use the within-bucket (max - min). Previously every series used max-min, which undercounts the rate of delta-temporality counters. - query_metrics now populates histogram_summary with the explicit bucket layout (bounds + element-wise-summed bucket_counts) alongside count/sum/min/max, so histogram metrics return their real distribution instead of only a misleading synthetic mean. This enables correct quantile reconstruction from the buckets. Both behaviours are covered by new live-ClickHouse integration tests (query_metrics_rate_respects_temporality, histogram_summary_aggregates_buckets).
…plar columns Adds a decode->store->raw-read test that inserts a metric carrying the nested Array(Tuple(...)) columns (exponential-histogram bucket counts, summary quantiles, and exemplars with trace/span ids) into a live ClickHouse container and asserts they survive. These RowBinary nested-tuple codepaths were previously only unit-tested at the row-mapping tier, never against a real ClickHouse server.
…e-count) histogram_summary previously summed raw histogram_count / bucket_counts across all rows in a window. For CUMULATIVE histograms re-exported multiple times (the OTLP default), each export is a running total, so summing them multiplied the counts by the number of exports — a live demo showed count=300 for 50 observations exported 6 times. Compute histogram_summary from a per-series sub-aggregation: each series (attributes_hash) is collapsed first (CUMULATIVE -> latest snapshot via argMax/max; DELTA or unspecified -> sum across the window), then summed across series up to the requested grouping granularity, matched back to the scalar rows by (bucket_ms, series_values). Scalar/quantile aggregations are unchanged. Covered by a new live-ClickHouse test: cumulative re-exports collapse to per-series latest, then sum across series.
Regenerate the SDK against the Phase C backend (typed aggregation / metric_type / label_filters params; histogram_summary, quantiles, series_key on MetricBucket). MetricsExplorer: - Default view now shows ALL metrics as an overview grid (one mini chart per metric with its latest value); click a card to drill into the detailed view. - Send the real aggregation + label_filters params. For histogram metrics, percentiles are computed client-side from the histogram_summary buckets (the backend scalar quantile runs over the synthetic mean). - Add a histogram Distribution panel (count/mean/p50-p99 + per-bucket bars). - Fix an infinite refetch loop: end_time used new Date() every render, changing the query key each render; the time bounds are now memoized.
Per-project saved dashboards persisted in Postgres (metric_dashboards table) as a typed JSON layout of sections -> metric tiles. Full CRUD under /api/otel/dashboards following Handler->Service->Data: - temps-entities: metric_dashboards entity + migration. - temps-otel: MetricDashboardService (CRUD, typed DashboardLayout/Section/Tile, aggregation + size/length-bounds validation), handlers (utoipa, permission_guard OtelRead/OtelWrite, audit-logged writes), routes + OtelApiDoc registration, DashboardNotFound -> 404. - get/update/delete are scoped by project_id (defense-in-depth against cross-tenant IDOR: a mismatched project_id returns 404, never another project's dashboard). A corrupt stored layout is logged, not silently swallowed. - 8 service tests (CRUD + validation + pagination cap). Frontend: Dashboards list, dashboard view (sections of metric-chart tiles, reusing the metrics-explorer chart + client-side histogram percentiles, with memoized time bounds), and a builder (add/rename sections, add tiles by metric + aggregation, save). SDK regenerated. Verified end-to-end against a running server + ClickHouse: CRUD round-trips the layout losslessly, IDOR is blocked (404 on mismatched project_id), and the UI renders saved dashboards with live charts. Follow-ups: domain-prefix the nested layout schema names (OtelDashboardLayout etc.) and add handler-layer 401/403 tests.
Drop the mx-auto/max-w container constraint on the metrics explorer, dashboards list, dashboard view, and builder so they use the full content width (e.g. ~1600px instead of a centered 1152px on wide screens) — more columns in the metric overview grid and more room for tile charts.
Collapse the separate "OTel Metrics" and "Dashboards" sidebar entries into a single Metrics surface with a route-backed segmented control: - Explore (index, /metrics): the all-metrics overview + per-metric drill-in. - Dashboards (/metrics/dashboards/*): the saved-dashboards list/view/builder, nested unchanged so their relative navigation is preserved. /dashboards/* now redirects to /metrics/dashboards. One nav entry; explore and curate live in the same place.
Add first-class threshold alerting on OpenTelemetry metrics. Alert rules attach directly to a metric (not to a dashboard), so the metric is the source of truth and any surface — explorer, dashboards — merely displays them. Backend (temps-otel): - metric_alert_rules entity + migration (project-scoped: name, metric_name, aggregation, comparator, threshold, window/for-duration, severity, enabled, last_state/last_value for firing-state tracking). - MetricAlertService: project-scoped CRUD with IDOR-safe by-id access (get/update/delete 404 on project mismatch), threshold finiteness + bounds validation, paginated list. - MetricAlertEvaluator: background tokio-interval evaluator. Queries the latest closed bucket (limit 2 to skip the in-progress one), derives the rule value per aggregation (incl. client-side histogram_quantile for percentile rules), runs a for-duration state machine, and fires/resolves through the existing temps_monitoring AlarmService so alerts reuse configured notification channels. No-data ticks preserve prior state. - CRUD handlers under /otel/alerts with audit logging. Frontend (web): - Alerts tab in the unified Metrics surface (Explore | Dashboards | Alerts). - MetricAlerts list with firing-state badge + one-line rule summary, MetricAlertForm create/edit, AlertsRouter. - Explorer overlays a rule's threshold as a reference line on the metric chart (critical=poor tone, else warn). - SDK regen for the new /otel/alerts endpoints. Verified end-to-end against a live ClickHouse-backed server: CRUD, IDOR (get/delete 404 on wrong project), threshold validation, and a breaching rule transitioning unknown→firing with the alarm actually fired. 18 unit tests pass; clippy -D warnings clean.
…ules
Reshape metric_alert_rules so future detector families (anomaly, EWMA,
forecast, outlier, Watchdog-style auto-watch) ship as code-only changes —
never another migration. Done now, before the table merges, so there is no
backfill or transition dance.
Schema: replace the static-only `comparator`/`threshold` columns with a
coarse `detection_kind` string discriminator + a typed-in-Rust, jsonb
`detection_config`. The cross-cutting eval envelope (metric, aggregation,
window, for-duration, severity, enabled, last_* state) stays typed columns;
only the detector-specific knobs move into the blob. Folded directly into the
in-flight create migration (zero rows, no ALTER).
detection_config is a serde internally-tagged enum (`DetectionConfig`) in the
new `temps_otel::detectors` module — copied verbatim from the sanctioned
`ProviderConfig`/`revenue_integrations.config` precedent: NO
`#[schema(discriminator)]` (a compile error with serde(tag) in utoipa 5.5.0).
The raw `serde_json::Value` lives only on the sea-orm column; every service
and DTO layer is fully typed, so the generated SDK is a usable TS
discriminated union — `(StaticParams & { kind: 'static' }) | …` — not `any`.
Today only `static` is evaluable; anomaly/forecast/outlier/auto_watch are
typed, schema-present stubs that validate() rejects at create time. Enabling
each later = a new validate arm + evaluator branch + openapi-ts regen.
`detection_kind` is a plain string (not a PG enum) so new kinds need no
ALTER TYPE either.
Evaluator now decodes the typed detector and branches on it (static =
`Comparator::breaches`); the bad-input surface moves to serde (unknown kind /
bad comparator / missing threshold -> 422 at deserialize) which is stronger
than the old string allowlists. Frontend maps the static form to/from
detection_config and the explorer/list narrow on `kind`.
Verified live (ClickHouse-backed): create/get round-trip the typed config
through jsonb; anomaly -> 400 (not yet supported); bad input -> 422; the
static evaluator still fires. 25 unit tests pass; clippy -D warnings clean;
SDK regenerated; frontend typechecks.
Make the `anomaly` detector evaluable end to end — it was a typed,
creation-rejected stub. A rule now learns a baseline band from history and
fires when the current value deviates from it, reusing the same for-duration
state machine and AlarmService as static threshold rules.
Detection math (pure, unit-tested) in `detectors`:
- robust_band = median + MAD·1.4826 (consistent-with-σ scale).
- anomaly_breaches = direction-aware (above/below/both) z-score test, with a
MIN_BAND_SCALE floor so a flat baseline can't divide by zero.
- season_cell buckets a timestamp into none/hourly/daily/weekly cells.
- validate() now accepts anomaly (robust/basic); agile/ewma and bad
hyperparameters (deviations≤0, pct∉(0,1], lookback∉1..=90) are rejected.
Evaluator branch (`metric_alert_evaluator`):
- Baseline fetched via the SAME query_metrics aggregation path as the scored
point (so counter-rate / histogram-percentile compare like-for-like — NOT
get_metric_baseline, which bypasses aggregation), cached per rule for 1h.
- Seasonal-cell filter with cold-start fallback to the global band; an
insufficient (<8 samples) or degenerate (flat) baseline PRESERVES state
rather than firing — no spurious alerts on thin history.
- fire() refactored to a detector-agnostic FireDetails (static vs anomaly
message/metadata, e.g. "820ms is 4.2σ from the baseline 210 ± 90").
- run_cycle prunes breach-timer + baseline caches for disabled/deleted rules
(also fixes a pre-existing breach_start leak).
Latent bug fixed (affected static AND anomaly): translate_bucket_interval only
accepted space-separated forms, so the evaluator's `format!("{}s", secs)`
("300s") silently fell back to INTERVAL 1 HOUR — every windowed query was
coarsened to hourly regardless of window_secs. Now also parses the compact
"300s"/"5m"/"1h"/"2d"/"1w" form.
Frontend: the alert form authors anomaly rules — a Detection selector swaps
the static comparator/threshold for algorithm / sensitivity (σ) / direction /
seasonality; the list summary reads the typed config.
Verified live (ClickHouse-backed): anomaly create accepted (was 400);
insufficient baseline preserves state; a normal value stays ok (no false
positive); an injected spike (100000 vs a ~100±15 band) transitions to firing
and raises an alarm. 31 unit tests pass; clippy -D warnings clean; frontend
typechecks; the form renders anomaly fields.
Anomaly detection was exposed but not explained. Three fixes: - History/eligibility banner: when a metric is picked for an anomaly rule, the form checks how much history it has and warns if it's under ~14 days — spelling out that the rule will sit at "unknown" and not alert until a baseline can be built (the silent-inert trap). The "Unknown" badge now also carries a tooltip explaining the same. - Sensitivity presets: the raw σ number is replaced with High/Medium/Low presets (2/3/4σ); the exact σ stays available under "Custom". - Advanced disclosure: algorithm + seasonality (sensible defaults most users won't touch) move behind an "Advanced" details block, leaving Sensitivity + Direction as the two primary knobs.
…tion) Editing an alert rule showed empty Aggregation + Detection selects and fell back to the static fields, regardless of the saved rule. Root cause: the form was created with placeholder defaults while the rule loaded, then updated via react-hook-form's `values` prop — and that reset drops Radix Select values that *change* during it (same-value selects like severity were unaffected, masking the bug). Fix: load the rule in a thin parent, then mount the form body once (keyed on the rule id) with the resolved `defaultValues` from the start, so no Select is reset post-mount. Verified: editing the anomaly rule now restores aggregation (max), detector (anomaly), sensitivity, direction, seasonality.
Add a read-only preview/backtest so an anomaly rule is legible before you save
it (and tunable after). The single highest-value affordance for a feature that
otherwise fails silently.
Backend:
- Extract a shared `BandModel` in `detectors` (per-seasonal-cell robust bands +
global fallback, built once and queried per timestamp). The evaluator's
`anomaly_eval` now uses it too, so the preview can never diverge from what
production would actually do.
- New `services::anomaly_preview` + `POST /otel/alerts/preview`: replays a
metric over a range against the band and returns per-bucket
{value, lower, upper, breaching} + breach_count + baseline sufficiency,
through the SAME query_metrics aggregation path as the evaluator.
Frontend:
- `AnomalyBacktest`: for an anomaly rule, calls the endpoint (debounced) and
shows "would have fired N× in the last 7 days" plus a chart of the value
against its expected band with breach markers, or an explicit
"not enough history" state when the baseline is thin.
- SDK regen for the new endpoint.
Verified live: backtest of the seeded anomaly rule reports 2 breaches over 7d
with band [55.5, 144.5] (median 100 ± 3·MAD); UI renders the count + chart.
31+ unit tests pass (incl. new BandModel test); clippy clean.
For an enabled anomaly rule on the charted metric, backtest its band over the visible range (the same preview endpoint the form uses) and shade the expected [lower, upper] region behind the line via a recharts ReferenceArea. Only shown when the explorer's aggregation matches the rule's (so the band sits on the same scale as the line) and the baseline is sufficient. ThresholdLineChart gains an additive `bands` prop; existing threshold lines are unchanged.
Turn the metrics pages from a neutral data browser into a problem surface: the system finds what's wrong and the user reads the answer. - Health header pinned above the tabs (Metrics.tsx): triaged status — firing alerts + active anomalies worst-first, with Alert/Warn/No-data/OK status dots and a firing count on the Alerts tab. Honest coverage states: "all systems healthy" vs "nothing is being watched yet" vs "couldn't load" — never false-green when nothing is monitored. - Status dots + toned line on overview cards (MetricsExplorer) and dashboard tiles (MetricTile): a firing metric reds out of a wall of green instead of looking healthy. Join keyed on (metric_name, aggregation) so a tile only reds for a rule that targets the series it shows. Redundant encoding (dot + tone + chip), never hue alone. - Severity sort: overview grid and the alerts list float worst-first (alert → warn → no-data → ok), so the 24-tile cap and the alert list stop hiding the broken thing. - Shared alert-status model (one cached listAlerts fetch) reused by all three. Fixes two real token bugs found in the review: - Added --success/--warning theme tokens (+ @theme mapping); badge.tsx used bg-success/bg-warning with no token, so the "OK"/healthy badge rendered with no background. - AnomalyBacktest used hsl(var(--primary)) — but the tokens are oklch(), so the band/line/breach dots painted transparent. Use bare var(--chart-1)/ var(--destructive) like the working chart. Frontend-only; typechecks clean.
"Did a deploy cause this?" is the first triage question, and Temps owns the deploy pipeline — a structural edge over Datadog. Overlay deploy events that fall inside the chart's visible window as distinct (purple, dashed) vertical markers, snapped to the nearest bucket (the categorical x-axis can't take a raw timestamp), labelled with the short commit hash. Scoped to the selected environment; timestamps normalised (sec or ms). ThresholdLineChart gains an additive `markers` prop (vertical ReferenceLine); existing lines unchanged.
…(Tier 2)
When a metric looks wrong the next question is "what else moved in this same
window?" — and Temps owns metrics, deploys, traces, and errors, so the answer
shouldn't require re-pivoting four tools by hand. Under the detail chart, a
"related signals" strip:
- Frames itself by live state: "This metric is firing — see what else
changed" when a rule on this (metric, aggregation) is firing, else a
neutral "What changed in this window". Reuses the cached listAlerts via
useAlertStatus — no extra fetch.
- Leads with the deploy answer ("1 deploy landed here — marked on the chart
above", or "No deploys in this window — rules out a release"), the literal
thing the chart's deploy markers visualise.
- Deep-links to Traces and Errors pre-scoped to the SAME window: Traces gets
range+env (which it already honours); Errors learns to read `?range=` so
the jump actually lands on that window (it widens the metrics-only 6h to
24h rather than ignore the intent). Plus a "Live view" jump to /observe.
Honesty: every link carries params the target page genuinely applies, and the
strip only states what it knows (deploys) — it never bluffs a trace/error
count it didn't fetch. Verified live: firing CPU-anomaly drill-in shows the
firing header + "1 deploy", and /errors?range=1h lands on "the last hour".
All curated-lucide icons (Network/Bug/Eye/Rocket/ArrowUpRight) confirmed in
the runtime bundle — the subset excludes ListTree/Telescope.
A long project name in the breadcrumb switcher (and long crumbs on deep paths like errors/<long-title>) wrapped the header to two lines because shadcn's BreadcrumbList is flex-wrap + break-words. Force the list to flex-nowrap with a min-w-0 ancestor chain, truncate + responsively cap every crumb (switcher label, intermediate links, and the current-page crumb — not just the switcher, or removing the wrap escape-hatch would overflow the terminal crumb under the action cluster), and clip at the breadcrumb boundary (overflow-hidden) with the right-hand action cluster pinned shrink-0. Verified by measurement: with every crumb forced to a 393px label, the breadcrumb stays 1 line and the header 64px at both 1280px and 375px, each crumb ellipsizes, and the breadcrumb never overlaps the action cluster.
Make "is anything on fire here?" answerable at a glance. A dashboard's status is the worst alert-rule status across the metrics its tiles plot — derived from the same cached listAlerts the per-tile dots already use, so no extra fetch and the signals can't disagree. The dashboards list shows a pulsing status dot + "N firing" per row; the dashboard view shows a firing badge in the header and a per-section count; "All clear" appears only when tiles are actually watched, nothing when no rule covers the dashboard (no vanity green). Hardened per adversarial review: - rollupStatus counts DISTINCT firing rules (Set of rule id), not tiles, so three tiles plotting one firing metric read "1 firing", not "3" (the name-fallback would otherwise map all three to the same rule). - ruleStatus + the firing/gathering lists now treat a disabled rule as not-firing: the backend freezes a disabled rule's last_state, so without this a monitor switched off mid-firing flashed a false red alarm. Fixed at the source, so the alerts/health surfaces benefit too. - the section "N firing" carries a severity title (no color-only meaning). Verified live: toggling the rule's `enabled` flips the dashboard between "1 firing" and "All clear" while last_state stays frozen-firing.
The project "Observe" sidebar group mixed OpenTelemetry signals with operational monitoring and carried a legacy "Metrics" (resource monitoring) entry that duplicated and added nothing over the OTel Metrics page. Split it: OpenTelemetry Observe · Traces · AI Traces · Metrics · Error Tracking Monitoring Uptime · Request Logs · AI Crawlers - "OTel Metrics" → "Metrics" (it's the only metrics surface now); "All events" → "Observe". - Removed the legacy project Metrics: dropped the nav entry, the `monitoring` route, and the ProjectMonitoring page component (used nowhere else). - Command palette: repointed its dead "Metrics" → /metrics and added "Observe", so removing the route leaves no broken command.
The PR's Changelog Check requires CHANGELOG.md to carry an [Unreleased] entry. Document the OTel metrics feature set: explorer, dashboards, alert rules with anomaly detection + backtest, deploy markers, cross-signal links, Datadog-style firing status, the OpenTelemetry/Monitoring nav grouping, the one-line header fix, the disabled-rule firing fix, and the legacy Metrics page removal.
Filtering traces by service threw ClickHouse Code 184 ILLEGAL_AGGREGATION: the trace-summary SELECT aliases `argMax(service_name, …) AS service_name`, which shadows the raw column, so an unqualified `service_name = ?` in WHERE resolved to the aggregate alias. Qualify it as `spans.service_name` so it binds the per-span column. Fixes both query_trace_summaries (Traces) and query_genai_trace_summaries (AI Traces); the count mirrors are qualified too to keep their filter SQL byte-identical. Not the space in the value — verified against live ClickHouse with "Observability Starter".
An orange line told you a metric was anomalous but the chart didn't show why. Datadog-style: overlay the detector's time-varying expected-range band and mark the points that left it. - ThresholdLineChart gains an optional `bandSeries` (LineChart → ComposedChart): two stacked Areas draw the [lower, upper] band behind the line; a stroke-less Line with a custom dot marks only the breaching points in red (recharts' Scatter plots null points, so it can't mark a sparse subset). Existing usages (tiles, web vitals) are unchanged. - Drill-in: backtest the rule's detector with the DISPLAYED aggregation (not the rule's) so the band always tracks the visible line and shows even when you're viewing a different aggregation than the rule alerts on. Per-bucket band values are merged onto the chart points (nearest-timestamp), replacing the old flat, aggregation-gated median band that almost never showed. Verified live: anomtest.cpu drill-in shows the expected band + a red breach dot at the spike; non-anomaly metrics and the web-vitals charts render unchanged.
Extract the drill-in's anomaly overlay into a reusable useAnomalyBand hook (find the enabled anomaly rule for a metric, backtest its detector over the visible window at the DISPLAYED aggregation, return a ThresholdLineChart bandSeries + a mergeBand that stamps per-bucket band/breach onto chart points). Wire it into MetricTile, so a metric shown standalone — dashboard tiles — gets the same expected-range band + breach markers as the explorer drill-in. Only the metrics that actually have an anomaly rule draw a band; everything else is unchanged (mergeBand is a no-op without a band).
Add a "Custom range…" option to the time selector that reveals a date-range picker; the absolute window is driven by ?start/?end in the URL alongside the relative ?range presets (the backend metric query already takes arbitrary start/end). A custom span derives its ClickHouse bucket interval from its width (bucketForSpan), mirroring the preset thresholds. Picking a preset clears the custom range and vice-versa. The anomaly band + deploy markers track the custom window too.
Alert emails were text-only. Now a fired metric alert leads with a rendered chart
of the metric's recent series — the line, the breach threshold (dashed, breach
side shaded + labelled e.g. "< 5.0") or the anomaly's expected band (amber), the
firing value marked with a red line + label ("last 5m max: …"), the evaluation
moment marked, and y-axis labels. Matches Datadog's alert chart.
No change to the shared NotificationData struct:
- The evaluator renders a self-contained inline SVG (no deps) and carries it on
the fire's existing `metadata` under a reserved `_chart_svg` key, with the
metric/value/threshold detail fields.
- AlarmService now propagates FireAlarmRequest.metadata into the notification (it
was dropped before) — so the email also gains a Details table of the context.
- The email renders `_chart_svg` raw + the other fields as rows; Slack, webhook,
and the persisted row skip `_`-prefixed keys so the SVG never leaks there.
Verified live: a fired alert's Mailpit email embeds the chart; existing
alarm/disk emails unaffected.
The anomaly alert email chart had no x-axis, so the time window was ambiguous. Thread bucket timestamps into AlertChart and render three ticks: a dated start (e.g. 'Jun 27 08:56'), a bare HH:MM midpoint, and a UTC-marked end at the evaluation moment. Bottom margin widened to fit.
Deterministic plain-language humanizer as the OSS default (no AI, no latency, no key), with an optional best-effort AI enricher over the existing temps-ai-gateway that only augments — never gates or delays — the deterministic text. Draws the OSS (one-shot summary) vs EE AI SRE (deep investigation) boundary.
Move the AI foundation out of temps-core into a dedicated, dependency-light temps-ai crate so any crate can use the trait + schemas without pulling in providers/HTTP. Adds: - schemas: reusable structured-output types (ErrorDiagnosis + FailureCategory) - diagnostics::diagnose_failure: one-call build/deploy failure diagnosis (log-tail bounded, schema-typed) for the deployer / UI to reuse temps-ai-gateway and temps-otel now depend on temps-ai (temps_core::ai -> temps_ai). temps-core drops the ai module + schemars dep.
A generic ai_conversations + ai_messages model keyed by polymorphic (context_type, context_id) — a resumable chat per interaction, deployment failures first. New temps-ai-chat crate over the temps-ai foundation; adds streaming/multi-turn to AiService; seeds via a per-context provider (deployment provider = diagnose_failure + job logs); SSE message endpoint replaying stored history through the gateway. OSS chat+explain (BYO key, opt-in); EE = proactive cross-signal AiSre. Reuses the dormant workspace_messages schema + Autopilot chat UI.
temps-ai gains ChatMessage/ChatTurnRequest/TokenStream + AiService::chat_stream (replays full history, streams the assistant reply token-by-token). GatewayAiService implements it over chat_completion_stream, parsing the OpenAI-SSE byte stream into text deltas.
…-023) Migration m20260627_000002: generic conversation store keyed by polymorphic (context_type, context_id) + append-only ai_messages (role/content, per-turn token/cost), FK cascade, context + ordering indexes. Adds projects.ai_debug_chat_enabled opt-in toggle. New sea-orm entities.
… (ADR-023) ConversationService: get_or_create (idempotent per context, seeds via the provider), find/history, archive, and streaming send_message (persists the user turn, replays full history through AiService::chat_stream, persists the assistant turn on completion). ConversationContextProvider trait (context_type + authorize + seed) lets each entity type seed its chat.
…R-023) DeploymentChatProvider seeds a chat from a failed deployment (state + cancelled_reason + failed jobs' error messages; log-tail enrichment TBD). AiChatPlugin builds ConversationService with the provider and mounts the chat routes (find / get-or-create / history / SSE send / archive), gated on ai_debug_chat_enabled + is_available. Registered in serve after the gateway.
…bled in project API (ADR-021/023) Thread both AI opt-in toggles through update_project_settings (request DTO, service signature + apply, handler call, test) and ProjectResponse (+ service Project struct + entity mapping) so they are settable + readable via the project API, ready for the settings UI switch.
…DR-023)
Adds typed client + react-query hooks for /projects/{id}/ai/conversations
(find/create/get/send/archive) and the ai_debug_chat_enabled /
ai_alert_summaries_enabled fields on the project response/update.
…oggles (ADR-021/023) Mirrors the attack_mode pattern: two switches (ai_debug_chat_enabled, ai_alert_summaries_enabled) in project security settings, saved via the project update mutation. Off by default; framed as using the user's own provider key/budget.
…ployments (ADR-023) Self-contained chat panel mounted on the deployment page only when ai_debug_chat_enabled is on AND the deployment failed. Finds/loads an existing conversation or starts one (auto-asks for a diagnosis), renders the turn history, and streams the assistant reply via a manual SSE fetch reader (the generated SDK can't stream). Defensive: degrades to an error message, never blocks the page.
DeploymentChatProvider now appends each failed step's log tail (via LogService, 2.5KB/step, line-trimmed) to the seed — the real diagnostic evidence. Adds AlertChatProvider (context_type=alert): seeds a chat to investigate a metric alert (metric, detection, severity, state, value, threshold/anomaly band) and make it actionable. Both registered in the plugin.
Extract a reusable <DebugChat contextType contextId> (replacing the
deployment-specific component) and mount it on the metric alert edit page
("Investigate with AI") so a firing alert becomes actionable — same streaming
chat, seeded from the alert context. Deployment page uses the generic
component too.
…DR-022) resolve_model required ai_gateway_config.allowed_models (no UI to set it), so AI stayed unavailable even with a key. Now falls back to a sensible default model for the first active provider key (gpt-4o-mini / claude-3-5- haiku / gemini-1.5-flash / grok-2), routed by prefix. Configure a key -> AI works.
…ject chat list (ADR-023) - AiService::chat() (non-streaming, tool-aware) + ChatTool/ToolCall/ChatTurnResponse; GatewayAiService maps OpenAI tools/tool_calls. - ConversationService runs an agentic tool loop when a context exposes tools (else plain streaming), and refreshes the system context from the provider on every turn so the model always sees current logs — not the stale creation-time snapshot. - DeploymentChatProvider gains a read_repo_file tool: reads files at the deployed commit via the configured Git provider API (base64-decoded, 16KB cap, errors returned as text), offered only for git-connected projects. Richer seed: pipeline overview + 6KB log tails. - New endpoints: per-project list + cross-project list_all (annotated with project name/slug) for the conversation switcher. - temps-ai-chat now depends on temps-git (optional GitProviderManager via DI) + base64.
… dock (ADR-022/023) - Settings → AI Providers page: configure OpenAI/Anthropic/xAI/Gemini keys (test, activate, replace, remove), backed by the existing provider-key API; nav entry added. - Persistent AI assistant dock rendered in the app shell (AiAssistantProvider + AiAssistantDock): a flex sibling that pushes content instead of a modal backdrop, so the rest of the console stays interactive and the chat keeps streaming across navigation. - Unified cross-project conversation list with source identification (project + type) and 'View source' links; per-chat Reset (archive + restart) and Delete (archive) with a confirm dialog. - DebugChatPanel: run-once init (fixes duplicate turns / StrictMode double-send), in-bubble typing indicator, distinct user/assistant bubbles, markdown rendering, onConversationChange for reset. - Entry points (top-bar button, failed deployment stage, alert form) now open the shared dock via context; removed the old modal Sheet wrappers and the bottom debug card.
Resolve CHANGELOG conflict by keeping both sides' Unreleased entries (OTEL metrics + AI assistant from this branch; Cloudflare Email provider + build-prereq docs from main).
…ing, typing-bubble + tool-loop tests (ADR-023) - Authz: project_scope_guard! on all per-project chat handlers (find/list/create/get/send/archive) + deny_deployment_token! on global GET /ai/conversations — closes cross-project access via FullAccess deployment tokens (2 blockers). - read_repo_file: reject ../, ., absolute, and backslash path segments + percent-encode path per segment in github_provider (defense-in-depth) — closes repo path traversal. - deployment.rs: char-boundary-safe log_tail()/bound() — fixes UTF-8 slice panics on multibyte log/file content (tests genuinely straddle the cut). - service.rs: unit tests for try_tool_loop (tool-then-prose, error fallback, MAX_ROUNDS cap, empty-final-text) + path-validation + multibyte tests. - web/DebugChatPanel: failed sends drop the optimistic empty assistant turn; TypingDots gated to the active streaming turn — no perpetual fake typing bubble on error. - Verified: cargo clippy --all-targets clean; cargo test -p temps-ai-chat 12 passed.
- Read/archive handlers now gate on a toggle-only ensure_chat_enabled (403 when ai_debug_chat_enabled is off) so disabling the feature revokes access to existing chat content; create/send keep the toggle+AI check. - Global list excludes conversations from disabled projects; team-visible scoping documented (no created_by filter, matching instance-wide ProjectsRead + the dock copy). - Disconnect-safe persistence: send_message and try_tool_loop now drive AI streaming + accumulation + the final DB insert in a detached task, relaying tokens over an mpsc channel — a client SSE disconnect no longer orphans the user turn. - Regenerated the OpenAPI SDK (adds list/listAll conversation endpoints + GlobalConversationResponse) and migrated the dock + panel's non-streaming calls onto the typed client (raw fetch kept only for the SSE stream); removed hand-copied interfaces. - DebugChatPanel: initializing skeleton so resuming a chat no longer flashes the empty state. - Tests: ChatError->Problem mapping, service-layer DB/scoping (incl. disabled-project exclusion + public_id scoping), and GatewayAiService::resolve_model/default-model. Verified: cargo check 0 errors; cargo test 219 passed; web AI files tsc/eslint clean.
The dock's conversation list is already cross-project, so the entry point shouldn't be project-scoped. The top-bar ✨ button now renders on every page whenever an AI provider is configured (active provider key) and opens the global chat list; open() no longer requires a projectId (kept for the failed-stage/alert initial-context opens). Resolves 'why isn't the AI button visible globally'.
… in the switcher (ADR-023)
- Backend: new ChatStreamEvent enum; send_message/try_tool_loop stream Token + ToolCall{id,name,arguments} + ToolResult{id,name,content}; handler maps them to SSE events (event: tool_call / tool_result, compact single-line JSON). Live-only (final assistant text still the only persisted turn). try_tool_loop tests updated; 27 pass.
- Frontend: DebugChatPanel renders each assistant turn's tool calls as collapsible cards (name + arguments + result) with max-h scroll + whitespace-pre-wrap/break-words so long args/results stay responsive in the dock; spinner while a tool runs.
- AiAssistantDock list rows now show the project favicon (Avatar + initial fallback) with a small context-type badge, so cross-project chats are identifiable at a glance.
- Verified live: read_repo_file emits tool_call + tool_result over SSE (fetched real package.json from the connected repo).
… back button (ADR-023) - Persistence: the agentic loop now records each executed tool (id/name/arguments/result) into the assistant message's metadata; MessageResponse exposes a typed `tools` field and getConversation returns them — so tool cards survive a page refresh (previously live-only). SDK regenerated (ToolInfo + MessageResponse.tools). - Per-message timestamps: each user/assistant turn shows a relative 'time ago'; TimeAgo now carries a title with the exact local date/time, so imprecise relative text reveals the precise timestamp on hover. - Dock header back control is now an icon-only chevron (title 'All chats') — the 'All chats' label was redundant. - Verified: cargo test 27 pass; re-fetching a conversation returns the persisted tool calls + results.
`temps migrate` ran the apply with no interrupt handling: pressing Ctrl+C killed the CLI but left the Postgres backend running the in-flight DDL (e.g. a slow CREATE INDEX) to completion. A re-run then started a SECOND backend migrating the same schema concurrently. - connect_for_migrate(): single persistent backend (pool of 1), tagged `application_name = 'temps_migrate'`, returns the backend PID. - cancel_migration_backend(): from a fresh connection, pg_cancel_backend() the captured PID (+ a sweep of any 'temps_migrate' session) so the DDL aborts and its transaction rolls back, leaving the DB at the last applied migration. - migrate.rs races the apply against tokio::signal::ctrl_c(); on interrupt it cancels server-side and exits with a clear "re-run to continue" message.
…by name The unified Observe feed listed every OTel span, burying the signal under internal child spans (executing api route, resolve page components, …). - fetch_spans now returns only ROOT spans (parent_span_id IS NULL) by default — one row per trace. The client-derived "Traces" count follows suit. - A name search drops the root restriction so child spans (SELECT guestbook, fetch POST …) remain discoverable, which is the point of grepping span names. - New migration m20260628_000001: a partial index idx_otel_spans_root_project_start (project_id, start_time DESC) WHERE parent_span_id IS NULL, so the root-only feed reads exactly the rows it returns (EXPLAIN confirms the planner uses it). Time-bounded name search stays on the existing (project_id, start_time) index — no trigram GIN needed.
… otel impl) Introduce a minimal read contract over the distributed-trace store so read-only consumers depend on temps-core, never on the heavy temps-otel storage crate (no cycle, mirrors how callers use dyn AiService instead of the gateway). - temps-core::traces: TraceReader trait + flat, dependency-light DTOs (TraceQueryFilter, TraceSummaryDto, TraceSpanDto, …); enums rendered as strings. - temps-otel: impl TraceReader for OtelService (maps the rich TraceQuery / TraceSummary / SpanRecord types onto the DTOs, clamps limits, normalises the root parent), registered as Arc<dyn temps_core::TraceReader> in the plugin.
…curity Bundled because these share the same conversation handler/service/plugin files. Trace tools: list_traces / get_trace ChatTools backed by dyn TraceReader, merged into every chat context (deployment, alert, project) — bounded, UTF-8-safe, project-scoped. Salvage a final answer from gathered tool results when the loop hits the round cap instead of answering blind. Project chats: new ProjectChatProvider (context_type "project") for general, project-scoped chats; "+ New chat" picker in the dock; multiple threads per project (uuid context_id) created lazily on the first message. Security hardening (write path): create/send now require ProjectsWrite (not ProjectsRead); input length bounds on context_type/context_id/content; audit logging for create/send/archive (best-effort, identifiers only); cap the global conversation list query; refactor ensure_chat_enabled to be DB-testable + add toggle-gate / length-bound tests.
dviejokfs
added a commit
to bherila/temps
that referenced
this pull request
Jun 30, 2026
…arity Bring the default TimescaleDB OTEL-metrics backend to parity with the ClickHouse backend (PR gotempsh#158): persist and query the full decoded metric record, with efficient day-one indexing. - batch_insert_metrics: store all 31 columns (start_time, temporality, is_monotonic, flags, description, exponential-histogram fields, summary quantiles, exemplars) as bound params. - query_metrics: temporality-aware histogram reconstruction (delta = element-wise bucket-count sum via WITH ORDINALITY; cumulative = latest snapshot per series), NULL-safe group_by join (IS NOT DISTINCT FROM), and Prometheus-style histogram quantile interpolation (p50/p95/p99) instead of a mean fallback. - Migration m20260629_000001: nullable columns + composite (project_id, metric_name, service_name, timestamp DESC) btree + GIN (attributes jsonb_path_ops) indexes; idempotent, reversible, verified safe on compressed hypertables. - Reads stay on the raw hypertable, NOT the 1min/1hr continuous aggregates: the CAGGs lag materialization and would silently drop recent buckets from charts. Raw + composite index + chunk pruning is correct and fast. Tests: 4 unit (histogram_quantile) + 11 integration vs real TimescaleDB (aggregations, label filter, group_by, histogram summary, delta element-wise sum, cumulative latest snapshot, quantile interpolation, NULL-label group_by, rate-per-sec, compressed-chunk DDL safety, bad-label rejection).
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a ClickHouse-first, full-fidelity OpenTelemetry metrics path to
temps-otel, alongside the existing TimescaleDB one. WhenTEMPS_CLICKHOUSE_*is configured, OTLP metrics are decoded losslessly and stored in a nativemetricsReplacingMergeTree; queries,list_metric_names, and the anomaly-detector helpers run natively against ClickHouse. The default (no-ClickHouse) install and theservice_metricsalerting bridge are unchanged.Context: the premise was "we support traces/spans but not metrics" — in fact metrics were already wired on TimescaleDB but flattened. This PR is the ClickHouse-first leg (CH is far better suited to high-cardinality metric labels + native quantiles), with TimescaleDB parity deferred.
What's included
MetricPointcarries temporality,is_monotonic,start_time, exponential-histogram buckets, summary quantiles, exemplars (trace/span), flags, description, and typed labels. Syntheticvalue=sum/countretained for histograms (the anomaly detector reads it).migrations/clickhouse/0003_metrics.sql(metricstable) + nativestore_metrics, replacing the delegate-to-Timescale stubs.ChMetricRowfield order is guarded against the RowBinary positional-serialization landmine by a DDL-parsing unit test.query_metrics(time-bucket, label filters, group-by, avg/sum/min/max/count/rate/quantile), parameterized + allowlisted (no injection surface). Store-neutral DTOs frozen so TimescaleDB can later satisfy the same contract.Metrics.tsx/MetricsExplorer.tsxmirroring Traces, wired to the generated SDK (no hand-rolled fetch), routed under the project + a sidebar entry.Verification (live ClickHouse, not skipped)
cargo test -p temps-otel: 263 lib + 4 CH-storage + 1 decode→store fidelity + 7 e2e + 15 TimescaleDB — all pass. The ClickHouse tests genuinely start a container and assert store→query→read-back of temporality,is_monotonic, histogram buckets, labels, and multi-series grouping.cargo check --lib -p temps-otel -p temps-migrationsis clean.Bugs found by the live tests + adversarial review (all fixed)
ReplacingMergeTreeORDER BY excluded labels and was second-resolution → distinct label-series (e.g.http.method=GETvsPOST) silently collapsed into one row. Fixed via aMATERIALIZEDlabel-hash + fulltimestampin ORDER BY.toUnixTimestamp64Milli(toStartOfInterval(...))(DateTime, not DateTime64). Fixed totoInt64(toUnixTimestamp(...))*1000.min/max/sum/quantileoverNullable(Float64)→ wrapped inassumeNotNull.message_on_stdout) never matched and the tests silently skipped (false green); switched to an HTTP/pingwait (http_wait_plain) + non-empty CH password.Deferred (tracked follow-ups, not in this PR)
rate()does not branch delta-vs-cumulative (treats all as max−min).histogram_summary=None) — misleading; no caller warning yet.Array(Tuple(...))columns are only unit-tested, never inserted into live ClickHouse.metrickind is off until then.Note
A pre-existing sibling has the same silent-skip testcontainer wait bug:
temps-analytics-events/src/services/clickhouse_backend.rs— its ClickHouse integration tests have also never actually run. Out of scope here; fix tracked separately.🤖 Generated with Claude Code