Repository navigation
feat(observability): unified Observe page (cockpit + console) - #71
Merged
Merged
Conversation
…tion, message extraction
Adds a project-level "Observe" surface that merges request logs, OTel
spans, error events, and revenue events into a single time-ordered
stream. Introduces a new `temps-observability` crate that exposes a
discriminated `ObservabilityEvent` wire shape (one row carries
everything the panel needs) and a per-kind parallel-fetch + k-way
merge service.
Wire correlation: new migration `m20260502_000001_add_observe_correlation`
adds nullable `trace_id`/`error_group_id`/`deployment_id`/
`environment_id`/`trace_id_indexed` columns across `proxy_logs`,
`revenue_events`, and `error_events` so the unified view can join an
event to its peers without follow-up queries. `temps-proxy` parses
W3C `traceparent` headers at ingest; `temps-error-tracking` promotes
`data.{sentry,}.contexts.trace.trace_id` to a top-level indexed
column.
Fix "Error: Unknown error" titles: error groups created from Sentry
SDK `captureMessage()` calls were stored with that literal title
because `exception.values[].value` is empty for message events. The
ingestion pipeline now probes `logentry.formatted` /
`logentry.message` / `message` / `exception.values[0].value` /
`breadcrumbs.values[0].message` / `extra.message`, and the same
fallback runs at read time in the Observe row mapper so legacy rows
also render their real text.
- `temps-observability`: new crate (service, handlers, filters,
cursor, plugin, OpenAPI types). 38 lib tests + integration tests
cover merge ordering, kind filtering, cursor stability, truncation
helpers, and per-kind fetcher correctness against TestDatabase
- W3C traceparent extractor in `temps-proxy::proxy` with 6 unit tests
(valid / missing / all-zero invalid / wrong length / non-hex /
case-folding)
- `extract_trace_id_from_data` helper in `temps-error-tracking` with
7 unit tests covering all three known JSONB layouts plus invalid
variants
- Heavy fields truncated server-side with `*_truncated` flags
(5 stack frames, 20 span attrs, header whitelist of ~10 keys)
…panel
Adds `/projects/:slug/observe` as the unified observability surface
for a project. Two-tier layout:
1. Cockpit header with one mini sparkline per kind (Requests,
Traces, Errors, Revenue) over the selected time range. Each card
is clickable to toggle that kind on/off. Traces default OFF
because `otel_spans` is a high-volume hypertable; the user opts
in via the cockpit card (dashed border + `'off'` label).
2. Console stream — terminal-style monospace rows with a color-coded
gutter per kind (sky=requests, violet=traces, rose=errors,
emerald=revenue), `HH:mm:ss` timestamps, and one-line summaries.
High density, scannable.
Side panel renders entirely from the row payload — no follow-up
fetch in the common case. URL-state filters so the page is
shareable. Sidebar gets an Observe entry; legacy detail routes
(`traces/:id`, `errors/:groupId`) still work as panel deep-links.
- New `web/src/pages/Observe.tsx` with cockpit + console + bucketing
+ sparkline + truncation-aware row renderer
- New `web/src/components/observe/{types,useObserveQuery,
ObserveFilterBar,rows/{ObserveRow,RowParts},panels/ObservePanel}`
- Regenerated SDK from the new `temps-observability` OpenAPI surface
- ProjectDetail mounts `/observe/*` and Sidebar surfaces "Observe"
…on tests
The first version of `m20260502_000001_add_observe_correlation` ran
plain Sea-ORM `ALTER TABLE … ADD COLUMN` against `proxy_logs`. Prod
runs `proxy_logs` as a TimescaleDB hypertable with a 7-day
compression policy from `m20260225_000001_add_proxy_logs_retention`,
and once chunks compress the ALTER fails with `chunk not found`,
leaves the schema half-applied, and Sea-ORM has no clean way to
retry. Local dev never caught it because nothing in our test rigs
had compressed chunks (rows weren't 7 days old yet).
This rewrite makes the migration safe in every prior state:
* Fresh installs (no extension installed)
* Production-style installs (compressed hypertable + active policy)
* Partially-failed prior runs (any subset of columns/indexes)
* Already-fully-applied installs (re-runs are no-ops)
* Installs where `proxy_logs` was never converted to a hypertable
Implementation: one PL/pgSQL `DO $$ … $$` block built entirely from
idempotent operations — `ALTER TABLE … ADD COLUMN IF NOT EXISTS`,
`CREATE INDEX IF NOT EXISTS`,
`remove_compression_policy(…, if_exists => TRUE)`,
`decompress_chunk(c, if_compressed => TRUE)`,
`add_compression_policy(…, if_not_exists => TRUE)`. Guarded by an
extension probe (`pg_extension WHERE extname = 'timescaledb'`) and a
hypertable probe (`timescaledb_information.hypertables`) so it
degrades cleanly on installs that lack either.
Operators upgrading from a binary that hit the original failure can
now boot the new binary directly — no manual SQL recovery needed.
Adds two regression tests in `tests/migration_tests.rs`:
* `test_observe_correlation_migration_handles_compressed_proxy_logs`
seeds 30 rows older than 7 days, force-compresses every eligible
chunk via `compress_chunk()`, then runs `Migrator::up()` and
asserts (a) every column + index exists, (b) every chunk is now
decompressed (catches regressions that skip the decompress step),
(c) the compression policy is restored, and (d) all 30 rows
survived. Verified to FAIL when the decompress block is removed.
* `test_observe_correlation_migration_is_idempotent` runs the
migrator, deletes the migration's row from `seaql_migrations`,
then runs the migrator again — must succeed without error.
Both tests use `testcontainers` and complete in ~6s.
Operators upgrading from earlier builds can hit Sea-ORM's load-phase
panic:
Migration file of version 'X' is missing, this migration has been
applied but its file is missing
This happens whenever a release squashes or removes a migration whose
row is still recorded in `seaql_migrations` from a prior install — the
loader scans the table, can't find a matching file, and refuses to
boot. The squash migration `m20260421_000001_squash_apr_post_v006`
explicitly notes operators must hand-INSERT its row, but that doesn't
address the orphaned predecessor rows.
Adds `cleanup_orphaned_migrations(&db)` and calls it from
`establish_connection()` immediately before `Migrator::up()`. The
cleanup:
* Skips entirely on a fresh install (no `seaql_migrations` table
yet — Sea-ORM creates it during `up()`)
* Builds the canonical version list from `Migrator::migrations()`
* Counts orphaned rows; when zero, returns silently to avoid log
noise on every startup
* Logs the orphan list at WARN level before deletion so any
surprise is visible in operator logs
* Deletes the orphan rows in a single parameterized DELETE — safe
because Sea-ORM only uses these rows to skip already-applied
migrations, and the actual schema state lives independently
Two integration tests in `tests/orphan_migration_cleanup.rs`:
* `cleanup_strips_orphans_so_migrator_can_load` reproduces the prod
failure: bring schema to current, INSERT four orphan rows,
confirm `Migrator::up()` now errors, run cleanup, confirm
`Migrator::up()` succeeds
* `cleanup_is_noop_on_clean_db` exercises both the missing-table
and clean-table paths to guarantee no false positives
Both run against a fresh `timescale/timescaledb-ha:pg18` testcontainer.
…igrator::up" This reverts commit eb7b21f.
…ate chunk-not-found race
The "chunk not found" prod error reproduces deterministically as a race
between the migration enumerating chunks (`show_chunks()` → operate on
each) and the background retention worker dropping chunks older than 30
days. By the time `decompress_chunk(stale_oid)` runs, retention has
already dropped that chunk → TimescaleDB throws.
Reproduced locally on `timescale/timescaledb-ha:pg18`:
INSERT 0 40
ERROR: chunk not found
DETAIL: schema_name: <null>, table_name: <null>
Fix:
1. Snapshot every active TimescaleDB job on `proxy_logs` (compression
+ retention policies). Pause them via `alter_job(scheduled => false)`.
2. Take `LOCK TABLE proxy_logs IN ACCESS EXCLUSIVE MODE` so any
in-flight worker finishes before we touch chunks, and no other
session can ALTER/INSERT until our migration commits.
3. Decompress chunks, run ALTERs, create indexes — race-free now.
4. Re-enable the jobs we paused.
Both protections are needed: the lock blocks new sessions, and the job
pause prevents Timescale's worker from starting new operations during
the brief window between our LOCK acquiring and our DDL completing.
Adds a third regression test
`test_observe_correlation_migration_survives_concurrent_retention` that
hammers retention in a background tokio task while the migration runs in
the foreground. Asserts the migration completes, the schema lands, and
every paused job is re-enabled. All three migration tests pass against
the fix in ~7s on `timescale/timescaledb-ha:pg18`.
…LETEs
Prod logs revealed the actual source of "chunk not found" — not the
Observe correlation migration but the OTel retention background task:
2026-05-02 09:20:14.462 ERROR: chunk not found
DETAIL: schema_name: _timescaledb_internal,
table_name: _hyper_15_3617_chunk
STATEMENT: DELETE FROM otel_metrics WHERE project_id = $1
AND timestamp < NOW() - INTERVAL '7 days'
The OTel hypertables (`otel_metrics`, `otel_spans`, `otel_log_events`)
all already have a native TimescaleDB
`add_retention_policy(..., INTERVAL '90 days')` registered in
`m20260225_000001_create_otel_tables`. Timescale enforces that policy
via `drop_chunks` — atomic, chunk-aware, race-free.
The app-level `apply_retention` in `TimescaleDbStorage` ran a parallel
`DELETE FROM otel_* WHERE timestamp < ...` on a tokio interval. The
DELETE's planner snapshots a chunk list, the native policy worker
drops one of those chunks, the DELETE's executor reaches the stale
chunk OID → `chunk not found`. The error then surfaced through the
retention task's logging (`error!(error = %e, ...)`) and got conflated
with migration failures in the rust-side error wrapper.
Fix:
1. `apply_retention` becomes a no-op (always returns 0). The trait
method is kept so callers/tests don't break, with a doc comment
explaining why and pointing at the native policy as the source of
truth.
2. The retention task in `temps-otel/plugin.rs` now discards the
immediate first `interval.tick()`. `tokio::time::interval` fires
at time zero by default, which would have hit the DB the moment
the plugin registered — exactly what we don't want during startup.
The retention task still exists as a hook for future per-project
logic; it just does no DB work today.
3. `retention_days` field on `TimescaleDbStorage` is annotated with
`#[allow(dead_code)]` since it's no longer read by the storage
layer (kept on the struct for config/API compatibility).
The existing `test_apply_retention_no_old_data` test is renamed to
`test_apply_retention_is_a_noop` and given a comment explaining the
contract — a regression that re-adds the DELETE would have to delete
this test to compile.
…migration Prior versions wrapped the column additions in a `decompress_chunk()` loop plus per-hypertable `CREATE INDEX`, both of which enumerate every chunk in `proxy_logs` (~280 in prod). That enumeration races with TimescaleDB's background `drop_chunks` retention worker — by the time we touch a chunk we just listed, it can already be gone, producing `chunk not found` and crash-looping the migrator on every server boot. Even with `alter_job(scheduled => false)` and `LOCK TABLE ... ACCESS EXCLUSIVE` the race window survived in prod logs. The minimum required for the Observe page is the columns themselves. `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` against a TimescaleDB hypertable is atomic across all chunks (no per-chunk DDL from us), so this strips: - the `decompress_chunk` loop - the job pause/restore dance - `LOCK TABLE` on `proxy_logs` - `CREATE INDEX` on `proxy_logs` and `revenue_events` (hypertables) The single `CREATE INDEX` left is on `error_events`, which is a regular (non-hypertable) Postgres table and therefore safe. Hypertable indexes become a separate operational task to run when the cluster is quiet.
…nts is a hypertable too
Prod log finally pinpointed the failing statement:
ERROR: chunk not found
DETAIL: schema_name: _timescaledb_internal, table_name: _hyper_3_1013_chunk
CONTEXT: SQL statement "CREATE INDEX IF NOT EXISTS idx_error_events_project_trace
ON error_events (project_id, trace_id_indexed)"
`_hyper_3_*` = hypertable id 3 = `error_events`. I had assumed it was a
regular Postgres table; it isn't. So `CREATE INDEX ON error_events`
enumerates chunks the same way `proxy_logs` and `revenue_events` do, and
races with `drop_chunks` from the retention policy → migration crash-loop.
Drop the index entirely. The Observe page works without it: error joins
go via `(project_id, occurred_at)` (already indexed by the hypertable
chunk dimension) plus `trace_id_indexed` as a filter. Chunk pruning
keeps it fast enough.
Index creation is now an out-of-band operational task to run when the
cluster is quiet — *not* something a startup migration should attempt.
Migration body is now 6 `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`
statements and nothing else. Zero chunk-touching SQL.
After repairing the prod orphan-chunk drift (caused by a prior `pg_dump`/`pg_restore` run without `timescaledb_pre_restore()`), the four hypertable indexes can be created cleanly. Putting them back into the correlation migration with `IF NOT EXISTS` so: - prod is a no-op (indexes already exist from manual operational fix) - fresh installs / dev / staging build them as part of normal migration The four indexes: - idx_error_events_project_trace (project_id, trace_id_indexed) - idx_proxy_logs_project_trace (project_id, trace_id) - idx_proxy_logs_error_group (error_group_id) WHERE NOT NULL - idx_revenue_events_project_occurred (project_id, occurred_at DESC) Header comment updated to document the orphan-chunk failure mode any operator might hit if they migrated their DB via raw `pg_dump`/`pg_restore` without the Timescale-aware wrappers — with a pointer that the cleanup is operational (manual detach + drop), not migration-level, because we will not silently delete user data from a startup migration.
The error_events entity gained a denormalized trace_id_indexed column for the Observe merge query; the make_error_event test helper was missing the field, breaking temps-agents lib test compilation. Also applies cargo fmt to migration_tests.rs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/projects/:slug/observe) that merges Requests, Traces, Errors, and Revenue into a single time-ordered timeline. Two-tier layout: cockpit header with one sparkline per kind (clickable to toggle) on top, terminal-style monospace stream below with color-coded gutters per kind. Side panel renders entirely from the row payload — no follow-up fetch in the common case.temps-observabilitycrate: discriminatedObservabilityEventwire type, per-kind parallel fetchers (Sea-ORM forproxy_logs/error_events/revenue_events, raw SQL for theotel_spanshypertable), k-way merge byts DESC, server-side truncation of heavy fields with*_truncatedflags. Traces default OFF (high-volume hypertable, opt-in via cockpit card). Runtime logs intentionally NOT included — they live on the dedicated Logs page.m20260502_000001_add_observe_correlationadds nullabletrace_id/error_group_id/deployment_id/environment_id/trace_id_indexedcolumns across the source tables.temps-proxyparses W3Ctraceparentheaders and stampsproxy_logs.trace_id;temps-error-trackingpromotesdata.{sentry,}.contexts.trace.trace_idto a top-level indexed column onerror_events.captureMessage()events: ingestion now probeslogentry.formatted/logentry.message/message/exception.values[0].value/breadcrumbs.values[0].message/extra.messagefor a usable string. Same fallback runs at read time in the Observe row mapper so legacy rows render their real text without a backfill.Test plan
cargo check --libpasses across the workspacecargo test --lib -p temps-observabilitypasses (38 lib + integration tests)cargo test --lib -p temps-proxy traceparent_testspasses (6 tests)cargo test --lib -p temps-error-tracking trace_id_extraction_testspasses (7 tests)bunx tsc --noEmitpasses inweb//projects/:slug/observeand confirm cockpit + console render with real datakinds/time_range/searchparams and is shareable