Repository navigation
feat(analytics): runtime-toggleable ClickHouse backend - #76
Merged
Merged
Conversation
dviejokfs
force-pushed
the
feat/clickhouse-analytics
branch
from
May 5, 2026 21:23
dfa3562 to
fe0944b
Compare
New temps-analytics-backend crate that will host the AnalyticsBackend trait separating handlers from storage engine choice. Phase 1 of the hybrid PG+ClickHouse plan: ships only the contract surface plus a TimescaleBackend skeleton and a feature-gated ClickHouseBackend stub. No behavior change yet — query methods migrate out of temps-analytics-events::services::events_service in subsequent commits.
Defines `AnalyticsEvents` trait covering all 12 read methods on `AnalyticsEventsService`. Service implements the trait via thin delegation to existing inherent methods — SQL stays in one place. Handler `AppState` now holds `Arc<dyn AnalyticsEvents>` for reads and keeps `Arc<AnalyticsEventsService>` (`events_writer`) for the single write path (`record_event`). This separates engine choice (read side, where ClickHouse will plug in via Phase 2) from the write fan-out (handled by the outbox in the hybrid plan, not by backend dispatch). Pure refactor: no SQL changes, 28/28 unit tests pass.
Phase 2 of the hybrid PG+ClickHouse plan. ClickHouse side: - Real `clickhouse` 0.13 client wired into `ClickHouseBackend` behind the `clickhouse` cargo feature (off by default — stock build remains a single static binary against PostgreSQL). - Working `health_check` via SELECT 1. - DDL migrations for `events` (ReplacingMergeTree, partitioned by month), `events_5m_mv` (SummingMergeTree pre-aggregation), and `sessions`. Schema mirrors the PG row shape so the outbox can fan out 1:1. Postgres side: - New `events_ch_outbox` table with FK ON DELETE CASCADE so project deletes propagate to CH without a separate tombstone path. Index on (delivered_at NULLS FIRST, enqueued_at) for the worker's scan. Worker: - `ChFanoutWorker` skeleton with shutdown signaling, configurable poll interval, batch size, and max attempts. The actual row mapping + ClickHouse insert is intentionally a no-op for now — landing it before the query-side SQL translation would risk shipping inserts that diverge from how reads consume them. Both ship together in the parity-tested follow-up. No behavior change for existing PG-only installs.
Phase 3 + 4 of the hybrid analytics plan.
ADR-012 documents the architectural decision: Postgres remains the
system of record, ClickHouse is an optional bring-your-own analytical
replica that operators point Temps at. Explicitly rejects making CH a
managed `ServiceType` in the providers framework — production CH is a
clustered system whose ops surface (Keeper, sharding, replication) is
out of scope for a PaaS.
Config:
- Adds `TEMPS_CLICKHOUSE_{URL,DATABASE,USER,PASSWORD}` env vars to
`ServerConfig`. All four must be present for CH to be considered
enabled (`is_clickhouse_enabled()`); partial config fails closed so a
half-configured operator never silently loses analytics.
- Updates 5 test fixtures that construct `ServerConfig` literals.
Default behavior unchanged: CH config defaults to None, fan-out worker
is not started, single PG-only binary still ships.
The earlier trait extraction modeled methods as long parameter lists, which conflated "what to ask the backend" with "how to render it as SQL." This commit fixes that by introducing per-method `*Spec` value types that capture the parameter space: - TimeRange + AnalyticsScope as shared sub-types (validation lives in the constructor: limit clamping, builder-style scope narrowing). - Twelve `*Spec` structs, one per trait method. - All `Clone + Debug` so they can be logged, fed into a parity test harness, and replayed against multiple backends. The `AnalyticsEvents` trait now reads as `query_xxx(&self, q: XxxSpec) -> Result<...>` which describes what to query, not how. A ClickHouse impl will consume the same `*Spec` values and render them via the typed CH client; the trait deliberately knows nothing about SQL. Naming: `*Spec` (not `*Query`) avoids collision with the existing HTTP request DTOs in `types::requests` which already use `*Query` for the JSON-shaped wire DTOs. Side fixes: - `PropertyBreakdownFilters` gains `Clone` so it can live in a Spec. - The `events_ch_outbox` migration drops its FK to `events.id` because Postgres can't FK to a TimescaleDB hypertable's composite unique constraint. Orphan handling moves to the worker (skip + mark delivered) which is the correct behavior when source events have been retention-dropped anyway. 28/28 unit tests pass. Pure refactor — no SQL changes, no behavior change, but the seam is now where `clickhouse-analytics-backend` will actually plug in.
End-to-end ClickHouse path. Setting all four TEMPS_CLICKHOUSE_* env vars
and rebuilding with --features clickhouse causes:
1. Plugin init applies the CH schema (events / events_5m_mv / sessions
tables, plus _temps_ch_migrations tracking) via a new embedded SQL
migration runner in temps-analytics-backend::migrations.
2. The ChFanoutWorker spawns on the tokio runtime. It claims undelivered
outbox rows with FOR UPDATE SKIP LOCKED, loads the events::Model
rows, maps them into ChEventRow (matching the CH DDL field-for-field),
batches them through clickhouse::Client::insert(), and marks the
outbox rows delivered. Orphans (events retention-dropped before the
worker drained them) are skipped without sending.
3. The read trait object becomes ClickHouseEventsBackend instead of the
PG-backed AnalyticsEventsService. All 12 query_* methods translate
the relevant *Spec into CH SQL using:
- count() / uniq() depending on aggregation level (HLL is ~1%
approximate, documented divergence from PG's exact COUNT DISTINCT)
- toStartOfInterval + WITH FILL for time bucketing (replaces
TimescaleDB's time_bucket_gapfill)
- FINAL on every read so ReplacingMergeTree merge-in-progress
state never surfaces duplicates
property_breakdown / property_timeline / dashboard_projects return a
clear Validation error pointing operators at PG fallback or the
contribution path. They depend on schemas that need their own design
pass.
4. record_event keeps writing to Postgres (system of record) and now
also enqueues each event_id into events_ch_outbox. The enqueue is a
separate statement, not in the events insert transaction — failure
to enqueue is logged at debug and does not fail the user request.
Side fixes:
- events_ch_outbox.event_id widened from integer to big_integer to
match events.id (i64).
- ClickHouseBackend gains client_clone() so the read backend and the
fan-out worker share one HTTP client.
- ChFanoutError::ClickHouseInsert.first_event_id widened to i64.
Default build (without --features clickhouse) is unchanged: single
binary against PG, outbox table populated but inert, no extra deps.
28/28 unit tests pass under both feature configurations. Workspace +
all tests compile clean, clippy clean.
What an operator does to enable CH:
export TEMPS_CLICKHOUSE_URL=https://ch.example:8443
export TEMPS_CLICKHOUSE_DATABASE=temps
export TEMPS_CLICKHOUSE_USER=temps
export TEMPS_CLICKHOUSE_PASSWORD=...
cargo build --release --bin temps --features clickhouse
./temps serve
The plugin logs "ClickHouse analytics backend enabled — applying
migrations and starting fan-out worker" on startup.
Validates the entire CH analytics pipeline against a live
clickhouse/clickhouse-server:24.8 testcontainer:
- Migration runner applies all 3 DDL files (events, events_5m_mv,
sessions) cleanly. Re-running is idempotent: report.applied is
empty on the second pass, all 3 marked skipped.
- ChEventRow shape round-trips through the typed Inserter and comes
back through the trait queries with correct values. If the row
derive ever drifts from the DDL, this fails loudly.
- Each implemented query_* method runs against the seeded fixture:
has_events, events_count (custom filter + project isolation),
event_type_breakdown, active_visitors (5-min window),
unique_counts (visitors / page_views / bad-metric Validation),
session_events (ordered ASC, None for missing),
events_timeline (WITH FILL clause executes — the trickiest CH SQL),
hourly_visits (page_view filter),
aggregated_buckets.
- Not-yet-implemented methods (property_breakdown,
property_timeline, dashboard_projects with non-empty input) return
Validation errors as documented, not silent wrong numbers.
- dashboard_projects with empty input short-circuits to empty without
hitting CH.
Per CLAUDE.md, the test skips gracefully when Docker is unreachable
(eprintln + return None) — never #[ignore]. When Docker is available
it runs end-to-end in ~60s.
29/29 tests pass under --features clickhouse, 28/28 on the default
build. The pipeline is now actually verified against a real CH
server, not just type-checked.
…time only The CH backend is now always compiled into every binary. Operators flip it on by setting `TEMPS_CLICKHOUSE_*` env vars on `temps serve`; no custom build is required. This is what users actually want: ship one binary, choose backends with flags. The previous feature flag created two distinct binaries — one that physically couldn't talk to ClickHouse — which is operationally worse and harder to support. Changes: - temps-analytics-backend: drop [features], make `clickhouse` crate a hard dep, drop all #[cfg(feature = "clickhouse")] from migrations and lib.rs. - temps-analytics-events: drop [features], hard dep on `clickhouse` crate, drop all feature gates from ch_fanout (no more dual constructor / no-op fallback), clickhouse_backend, mod.rs, and plugin.rs (no more dead cfg(not(feature)) warn-and-fallback path). The CH integration test under clickhouse_backend.rs now runs as part of the standard `cargo test` invocation. 29/29 tests pass with no feature flags, including the live testcontainer test against clickhouse-server:24.8 (skips gracefully when Docker is unavailable, per CLAUDE.md).
…-types Two bullets under [Unreleased] / Added: - runtime-toggleable CH backend (env vars, no rebuild, ADR-012) - AnalyticsEvents query value-types (*Spec structs)
…runbook
Closes the three release-blocking gaps from the post-merge review.
1. Implements the three previously-stubbed CH trait methods:
- query_property_breakdown: groups on plain CH columns; geo
(country/region/city) denormalized into the events table at
fan-out time so no cross-database join. Filter set mirrors PG
(drill-down by country/region/browser/os/channel/referrer); the
PG-only self-referral filter against project_custom_domains is
documented as a known divergence.
- query_property_timeline: same shape with toStartOfInterval bucket
dimension; auto-detects bucket size by range width to match PG.
- query_dashboard_projects: three CH queries (current period uniq
visitors, previous period uniq visitors, hourly sparkline) with
in-memory densification mirroring PG generate_series gap-fill.
Trend percentage calculation identical to Timescale impl.
2. CH events DDL gains country/region/city columns (LowCardinality
strings, empty = "no value" sentinel); fan-out worker enriches by
loading the matching ip_geolocations rows for each batch in a
single round-trip and passing them to row_to_ch.
3. Fan-out worker gains two periodic tasks alongside the poll loop:
- retention sweep (hourly): DELETE delivered outbox rows older
than 7 days. Without this the outbox grew unbounded.
- dead-letter scan (5 minutes): COUNT rows where attempts >=
max_attempts and warn-log when non-zero. Operators see the
problem in their log aggregator without polling the DB.
4. New how-to: docs/howto/enable-clickhouse-analytics covers
prerequisites, env vars, what happens on startup, three
verification queries, rollback path, what's-on-each-backend
matrix, runbook for outbox/CH-down/credential-rotation/backfill,
and known limitations.
Integration test updated to seed geo columns and assert real return
values for the three implemented methods. 29/29 tests pass (28 unit +
1 live CH testcontainer at ~60s).
…page The analytics feature page previously only described the PostgreSQL/TimescaleDB path. With CH support shipped, operators landing on /docs/analytics need to know: - ClickHouse exists as an opt-in read backend - When to use it (~50M events/month or slow dashboards) - How to enable it (four env vars, no rebuild) - Where the full runbook lives (/docs/enable-clickhouse-analytics) Adds a new "Storage Backend" H2 section between Overview and Visitor Analytics with a backend-comparison table and the env-var snippet. Links into the existing howto for the long-form runbook. Architecture bullets reworded to clarify PG is the system of record either way and CH is opt-in. No changes to the rest of the page.
dviejokfs
force-pushed
the
feat/clickhouse-analytics
branch
from
May 6, 2026 07:01
38f4d0b to
68508dd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Operators can now run ClickHouse as the analytics read backend in place of TimescaleDB by setting
TEMPS_CLICKHOUSE_*env vars ontemps serve. Same binary, no rebuild, no cargo feature flag.When CH is enabled the plugin:
_temps_ch_migrations).ChFanoutWorkeron the tokio runtime — claims undelivered outbox rows withFOR UPDATE SKIP LOCKED, batches them throughclickhouse::Client::insert(), marks rows delivered.Arc<dyn AnalyticsEvents>toClickHouseEventsBackend.Postgres remains the system of record.
record_eventwrites to PG synchronously and enqueues each event_id intoevents_ch_outbox; the worker drains asynchronously. CH outage never blocks ingestion.How to enable
Default behavior unchanged when env vars are unset: single binary against PG, outbox table populated but inert (so flipping the switch later replays history, modulo retention).
Trait shape
The
AnalyticsEventstrait describes parameters, not SQL. Each method takes a*Specvalue-type:Backends consume the spec and render however they want — Timescale builds parameterized SQL with
$1, $2; CH usesclickhouse::Client::query()with typed.bind(). Adding a third backend means implementing the trait, nothing else.What's implemented
10 of 12
query_*methods translated to CH dialect with full integration test coverage:query_events_count,query_session_events,query_has_events,query_event_type_breakdown,query_events_timeline,query_active_visitors,query_hourly_visits,query_unique_counts,query_aggregated_buckets,query_dashboard_projects(empty case)CH-side specifics:
count()/uniq()for aggregation levels (HLL is ~1% approximate, documented divergence from PG),WITH FILLfor time-bucket gap-fill,FINALon every read forReplacingMergeTreecorrectness.What's not implemented
3 methods return a clear
Validationerror rather than silent wrong numbers:query_property_breakdown— needs CH dialect forPropertyColumnJSON path accessquery_property_timeline— samequery_dashboard_projects(non-empty) — depends on TimescaleDB continuous aggregatesThese are bounded follow-up work, not blockers.
Tests
Live testcontainer test against
clickhouse/clickhouse-server:24.8:ChEventRowshape round-trips through the typedInserter.query_*methods validated against seeded fixture: project isolation holds,WITH FILLparses,FINALreads return correct counts, bad-metric inputs yieldValidationerrors, missing sessions returnNone.Validationerrors.Skips gracefully when Docker is unreachable (per CLAUDE.md: never
#[ignore]). 29/29 tests pass on the standardcargo testinvocation (~60s with Docker, ~12s without).Architecture decision
ADR-012 documents why CH is bring-your-own (operator points Temps at an existing cluster) rather than a managed
ServiceTypelike Postgres. Production CH means Keeper, sharding, replication — out of scope for a PaaS bundle.Commits
Test plan
cargo check --lib --testsworkspace-wide: 0 errorscargo test --lib -p temps-analytics-events: 29/29 pass (incl. live CH testcontainer)cargo clippy --lib -p temps-analytics-events: cleancargo clippy --lib -p temps-analytics-backend: cleantemps servewithoutTEMPS_CLICKHOUSE_*; verify analytics endpoints still hit PG.temps serve; verify migrations run, fan-out worker logs, an event written to PG appears in CH within a few seconds.