feat(observability): unified Observe page (cockpit + console)

CHANGELOG.md

@@ -8,7 +8,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
-- **CLI credential-based login (`@temps-sdk/cli`)**: `bunx @temps-sdk/cli login <url>` now authenticates with email + password (and TOTP MFA when enabled) instead of requiring users to mint and paste an API key. Internally hits a new `POST /auth/cli/login` endpoint that mints a 90-day API key scoped to the user's role. The previous fragile `/auth/login` + cookie-scrape + `/api/tokens` flow is replaced. `--api-key` and `--magic` login modes still work.
+- **Unified Observe page (`/projects/:slug/observe`)**: new project-level page that merges Requests, Traces, Errors, and Revenue into a single time-ordered timeline. Cockpit header shows one sparkline per kind over the selected time range — clickable to toggle that kind on/off. Below it, a console-style monospace stream with color-coded gutters per kind (sky=requests, violet=traces, rose=errors, emerald=revenue), `HH:mm:ss` timestamps, and rich one-line summaries. Side panel renders entirely from the row payload — no follow-up fetch in the common case. Filter state lives in URL search params so the page is shareable. Traces default OFF (high-volume hypertable, opt-in via cockpit card). Runtime logs are intentionally NOT included — they live on the dedicated Logs page because their volume would dominate the merged business-signal timeline
+- **`temps-observability` crate**: new merge service backing the Observe page. Per-kind parallel queries against `proxy_logs`, `error_events`, `revenue_events`, and the `otel_spans` TimescaleDB hypertable (raw SQL via `FromQueryResult`); k-way merged by `ts DESC` and trimmed to a single page. Heavy fields are truncated server-side (stack frames → first 5, span attributes → first 20 keys alphabetized, headers → whitelist of ~10 keys) with `*_truncated` flags so the panel can offer a "Show full" un-truncate fetch when needed. Wire shape is a discriminated `ObservabilityEvent` union — every row carries its own `type` discriminator and all data the panel needs
+- **Cross-source correlation columns**: new migration `m20260502_000001_add_observe_correlation` adds `proxy_logs.trace_id` + `proxy_logs.error_group_id` + indexes, `revenue_events.deployment_id` + `revenue_events.environment_id` + `revenue_events.trace_id`, and `error_events.trace_id_indexed` (denormalized from `data.trace.trace_id` JSONB for index speed). All nullable — old rows just render without correlation links. Lets the Observe view jump from any event to its peers in the same trace
+- **W3C `traceparent` extraction in proxy**: `temps-proxy` now parses inbound `traceparent` headers and stamps the 32-hex trace_id onto `proxy_logs.trace_id` at write time. Validates length, hex chars, and rejects the all-zero invalid trace_id reserved by the W3C spec. Six unit tests cover the happy path, missing/malformed/invalid inputs, and case-folding
+- **OTel trace_id promotion in error tracking**: `temps-error-tracking` ingestion now probes `data.sentry.contexts.trace.trace_id`, top-level `data.contexts.trace.trace_id`, and `data.trace.trace_id` and promotes the first valid 32-hex value to the indexed `error_events.trace_id_indexed` column. Seven unit tests cover all three layouts plus invalid lengths/chars and the all-zero case
+
+### Fixed
+- **Error group titles "Error: Unknown error"**: error groups created from Sentry SDK `captureMessage()` calls (which omit `exception.values[].value`) were stored with the literal title `"Error: Unknown error"`, masking the real message. The ingestion pipeline now probes the raw Sentry payload at `logentry.formatted`, `logentry.message`, top-level `message`, `exception.values[0].value`, the first breadcrumb message, and `extra.message` for a usable string. Same fallback runs at read time in the Observe row mapper, so existing groups also render their real text — group titles in the database remain as-is, but the Observe row and the error detail page now show the real message regardless. Future events will get correctly-titled groups at create time
+
+ `bunx @temps-sdk/cli login <url>` now authenticates with email + password (and TOTP MFA when enabled) instead of requiring users to mint and paste an API key. Internally hits a new `POST /auth/cli/login` endpoint that mints a 90-day API key scoped to the user's role. The previous fragile `/auth/login` + cookie-scrape + `/api/tokens` flow is replaced. `--api-key` and `--magic` login modes still work.
 - **CLI multi-context support**: `bunx @temps-sdk/cli context list | use | remove | current` lets one workstation stay logged into multiple self-hosted Temps servers simultaneously. Each context stores `{name, url, apiKey, email, keyPrefix, expiresAt}` in `~/.temps/.contexts.json` (mode 0600). The active context drives `apiUrl` and `apiKey` resolution for every other CLI command, with environment variables (`TEMPS_API_URL`, `TEMPS_API_TOKEN`, `TEMPS_API_KEY`) still taking precedence for CI / one-off overrides. Distinct from `temps instances`, which manages Temps Cloud VPS instances.
 - **CLI `temps logout` revokes server-side**: now calls `POST /auth/cli/logout` (best effort) to revoke the API key on the server before clearing local credentials. Pass `--local-only` to skip the network call when the server is unreachable, or `--context <name>` to log out of a specific context.
 - **CLI `temps whoami` surfaces context**: shows the active context name, key prefix, and expiry alongside the user info; `--json` includes the full context object.

Cargo.toml

@@ -74,6 +74,7 @@ members = [
     "crates/temps-preview-gateway",
     "crates/temps-external-plugins",
     "crates/temps-revenue",
+    "crates/temps-observability",
     "examples/example-plugin",
     "examples/lighthouse-plugin",
     "examples/indexnow-plugin",

crates/temps-agents/src/services/executor.rs

@@ -4002,6 +4002,7 @@ mod tests {
                     }
                 ]
             })),
+            trace_id_indexed: None,
             created_at: Utc::now(),
         }
     }

crates/temps-cli/Cargo.toml

@@ -30,6 +30,7 @@ temps-workspace = { path = "../temps-workspace" }
 temps-sandbox = { path = "../temps-sandbox" }
 temps-backup = { path = "../temps-backup" }
 temps-revenue = { path = "../temps-revenue" }
+temps-observability = { path = "../temps-observability" }
 temps-config = { path = "../temps-config" }
 temps-core = { path = "../temps-core" }
 temps-database = { path = "../temps-database" }

crates/temps-cli/src/commands/serve/console.rs

@@ -50,6 +50,7 @@ use temps_monitoring::{
     OutageDetectionService,
 };
 use temps_notifications::NotificationsPlugin;
+use temps_observability::ObservabilityPlugin;
 use temps_otel::plugin::OtelPlugin;
 use temps_projects::ProjectsPlugin;
 use temps_providers::ProvidersPlugin;
@@ -878,6 +879,13 @@ pub async fn start_console_api(params: ConsoleApiParams) -> anyhow::Result<()> {
     let revenue_plugin = Box::new(RevenuePlugin::new());
     plugin_manager.register_plugin(revenue_plugin);
 
+    // 11b. ObservabilityPlugin - unified Observe page (merges runtime logs,
+    // requests, spans, errors, revenue into one event stream). Read-only,
+    // no outbound calls; depends on the database only.
+    debug!("Registering ObservabilityPlugin");
+    let observability_plugin = Box::new(ObservabilityPlugin::new());
+    plugin_manager.register_plugin(observability_plugin);
+
     // AI Gateway Plugin - provides AI provider key management and OpenAI-compatible API
     debug!("Registering AiGatewayPlugin");
     let ai_gateway_plugin = Box::new(temps_ai_gateway::AiGatewayPlugin::new());

crates/temps-entities/src/error_events.rs

@@ -215,6 +215,12 @@ pub struct Model {
     #[sea_orm(column_type = "JsonBinary")]
     pub data: Option<serde_json::Value>,
 
+    // ===== Indexed correlation =====
+    /// Denormalized copy of `data.trace.trace_id` promoted to a top-level
+    /// indexed column so the Observe merge query can join error rows with
+    /// the originating request/span without a JSON probe.
+    pub trace_id_indexed: Option<String>,
+
     // ===== Metadata =====
     pub created_at: DBDateTime,
 }

crates/temps-entities/src/proxy_logs.rs

@@ -113,6 +113,16 @@ pub struct Model {
 
     /// Visitor ID for tracking unique visitors
     pub visitor_id: Option<i32>,
+
+    /// OTel trace_id stamped from the request span when available — lets the
+    /// unified Observe view join request rows with their child spans, runtime
+    /// log lines, and any captured exceptions.
+    pub trace_id: Option<String>,
+
+    /// Set when the request produced a captured exception in
+    /// `temps-error-tracking`; lets the Observe row deep-link to the group
+    /// without an extra query.
+    pub error_group_id: Option<i32>,
 }
 
 #[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]

crates/temps-entities/src/revenue_events.rs

@@ -30,6 +30,20 @@ pub struct Model {
     pub price_id: Option<String>,
     /// Opaque product reference (Stripe product_id, …). NULL mirrors `price_id`.
     pub product_id: Option<String>,
+
+    /// Deployment that produced this revenue event, when the SDK supplied
+    /// the `X-Temps-Deployment-Id` header. NULL for webhooks delivered
+    /// outside a request context.
+    pub deployment_id: Option<i32>,
+
+    /// Environment that produced this revenue event, when known via the
+    /// SDK header.
+    pub environment_id: Option<i32>,
+
+    /// OTel trace_id from the originating request, when supplied by the
+    /// SDK header. Lets the Observe view link a payment to the request that
+    /// triggered it.
+    pub trace_id: Option<String>,
 }
 
 #[derive(Copy, Clone, Debug, EnumIter, DeriveRelation)]

crates/temps-error-tracking/src/services/error_ingestion_service.rs

@@ -10,6 +10,71 @@ use temps_entities::{error_events, error_groups};
 
 use super::types::{CreateErrorEventData, ErrorTrackingError};
 
+/// Pull a 32-hex-char OTel trace_id out of an error event's JSONB `data`
+/// blob. Probes both the Sentry layout (`contexts.trace.trace_id`) and our
+/// internal layout (`trace.trace_id`). Returns `None` when no valid
+/// trace_id is present so the indexed column stays NULL for legacy events.
+fn extract_trace_id_from_data(data: &serde_json::Value) -> Option<String> {
+    fn validate(v: &serde_json::Value) -> Option<String> {
+        let s = v.as_str()?.trim().to_ascii_lowercase();
+        if s.len() == 32 && s.chars().all(|c| c.is_ascii_hexdigit()) && s.chars().any(|c| c != '0')
+        {
+            Some(s)
+        } else {
+            None
+        }
+    }
+
+    // Sentry / our internal wrapper: data.sentry.contexts.trace.trace_id
+    if let Some(v) = data.pointer("/sentry/contexts/trace/trace_id") {
+        if let Some(s) = validate(v) {
+            return Some(s);
+        }
+    }
+    // Top-level Sentry payload: data.contexts.trace.trace_id
+    if let Some(v) = data.pointer("/contexts/trace/trace_id") {
+        if let Some(s) = validate(v) {
+            return Some(s);
+        }
+    }
+    // Custom layout via TraceContext: data.trace.trace_id
+    if let Some(v) = data.pointer("/trace/trace_id") {
+        if let Some(s) = validate(v) {
+            return Some(s);
+        }
+    }
+    None
+}
+
+/// Probe a raw Sentry payload for a usable human-readable message.
+///
+/// SDKs that send `captureMessage()` (no exception) put the text in
+/// `logentry.formatted` (the rendered string after parameter
+/// substitution) or `logentry.message` (the template). SDKs sending
+/// `captureException()` put it in `exception.values[0].value`. Some
+/// older SDKs use the deprecated top-level `message`. We probe in
+/// priority order and return the first non-empty hit.
+pub fn extract_message_from_raw(raw: Option<&serde_json::Value>) -> Option<String> {
+    let raw = raw?;
+    let candidates = [
+        "/logentry/formatted",
+        "/logentry/message",
+        "/message",
+        "/exception/values/0/value",
+        "/breadcrumbs/values/0/message",
+        "/extra/message",
+    ];
+    for path in candidates {
+        if let Some(s) = raw.pointer(path).and_then(|v| v.as_str()) {
+            let trimmed = s.trim();
+            if !trimmed.is_empty() {
+                return Some(trimmed.to_string());
+            }
+        }
+    }
+    None
+}
+
 /// Service for ingesting and processing error events
 pub struct ErrorIngestionService {
     db: Arc<DatabaseConnection>,
@@ -313,34 +378,45 @@ impl ErrorIngestionService {
         error_data: &CreateErrorEventData,
         _fingerprint: &str,
     ) -> Result<i32, ErrorTrackingError> {
-        // Use first exception for title, or fall back to legacy fields
+        // Use first exception for title, or fall back to legacy fields.
+        // When `exception_value` is missing (most `captureMessage` payloads
+        // and Sentry SDKs that send only an event-level message), probe
+        // the raw event for a usable message so the title doesn't render
+        // as the useless literal "Error: Unknown error".
         let (exception_type, exception_value) =
             if let Some(first_exception) = error_data.exceptions.first() {
-                (
-                    first_exception.exception_type.clone(),
-                    first_exception
-                        .exception_value
-                        .clone()
-                        .unwrap_or_else(|| "Unknown error".to_string()),
-                )
+                let value = first_exception
+                    .exception_value
+                    .clone()
+                    .filter(|s| !s.trim().is_empty())
+                    .or_else(|| extract_message_from_raw(error_data.raw_sentry_event.as_ref()));
+                (first_exception.exception_type.clone(), value)
             } else {
+                let value = error_data
+                    .exception_value
+                    .clone()
+                    .filter(|s| !s.trim().is_empty())
+                    .or_else(|| extract_message_from_raw(error_data.raw_sentry_event.as_ref()));
                 (
                     error_data
                         .exception_type
                         .clone()
                         .unwrap_or_else(|| "Error".to_string()),
-                    error_data
-                        .exception_value
-                        .clone()
-                        .unwrap_or_else(|| "Unknown error".to_string()),
+                    value,
                 )
             };
 
-        let title = format!(
-            "{}: {}",
-            exception_type,
-            exception_value.chars().take(100).collect::<String>()
-        );
+        let title = match exception_value.as_deref() {
+            Some(v) if !v.trim().is_empty() => format!(
+                "{}: {}",
+                exception_type,
+                v.chars().take(100).collect::<String>()
+            ),
+            _ => exception_type.clone(),
+        };
+        // `exception_value` flows on into the embedding; fall back to the
+        // type so we still get a meaningful similarity vector.
+        let exception_value = exception_value.unwrap_or_else(|| exception_type.clone());
 
         // Create embedding from error message for similarity search (reuse exception_value from above)
         let embedding = self.create_embedding(&exception_value);
@@ -495,6 +571,14 @@ impl ErrorIngestionService {
                 )
             };
 
+        // Promote the OTel trace_id (if any) from the JSONB data blob to the
+        // top-level `trace_id_indexed` column so the unified Observe view can
+        // join error rows to their originating request/span without a JSON
+        // probe. Probes both the Sentry layout (`contexts.trace.trace_id`)
+        // and our internal layout (`trace.trace_id`); validates 32-hex
+        // before persisting.
+        let trace_id_indexed = extract_trace_id_from_data(&data_json);
+
         let new_event = error_events::ActiveModel {
             error_group_id: Set(group_id),
             fingerprint_hash: Set(fingerprint.to_string()),
@@ -508,6 +592,7 @@ impl ErrorIngestionService {
             deployment_id: Set(error_data.deployment_id),
             visitor_id: Set(error_data.visitor_id),
             ip_geolocation_id: Set(error_data.ip_geolocation_id),
+            trace_id_indexed: Set(trace_id_indexed),
             created_at: Set(Utc::now()),
             ..Default::default()
         };
@@ -537,3 +622,83 @@ impl ErrorIngestionService {
 mod tests {
     include!("error_ingestion_tests.rs");
 }
+
+#[cfg(test)]
+mod trace_id_extraction_tests {
+    use super::extract_trace_id_from_data;
+    use serde_json::json;
+
+    #[test]
+    fn extracts_from_sentry_wrapped_layout() {
+        let data = json!({
+            "source": "sentry",
+            "sentry": {
+                "contexts": {
+                    "trace": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" }
+                }
+            }
+        });
+        assert_eq!(
+            extract_trace_id_from_data(&data),
+            Some("4bf92f3577b34da6a3ce929d0e0e4736".to_string())
+        );
+    }
+
+    #[test]
+    fn extracts_from_top_level_contexts_layout() {
+        let data = json!({
+            "contexts": {
+                "trace": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" }
+            }
+        });
+        assert_eq!(
+            extract_trace_id_from_data(&data),
+            Some("4bf92f3577b34da6a3ce929d0e0e4736".to_string())
+        );
+    }
+
+    #[test]
+    fn extracts_from_internal_trace_layout() {
+        let data = json!({
+            "trace": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" }
+        });
+        assert_eq!(
+            extract_trace_id_from_data(&data),
+            Some("4bf92f3577b34da6a3ce929d0e0e4736".to_string())
+        );
+    }
+
+    #[test]
+    fn returns_none_when_missing() {
+        let data = json!({ "exception": "boom" });
+        assert_eq!(extract_trace_id_from_data(&data), None);
+    }
+
+    #[test]
+    fn rejects_invalid_lengths_and_chars() {
+        let bad = json!({ "trace": { "trace_id": "deadbeef" } });
+        assert_eq!(extract_trace_id_from_data(&bad), None);
+
+        let nonhex = json!({ "trace": { "trace_id": "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz" } });
+        assert_eq!(extract_trace_id_from_data(&nonhex), None);
+    }
+
+    #[test]
+    fn rejects_all_zero_invalid_trace_id() {
+        let data = json!({
+            "trace": { "trace_id": "00000000000000000000000000000000" }
+        });
+        assert_eq!(extract_trace_id_from_data(&data), None);
+    }
+
+    #[test]
+    fn lowercases_input() {
+        let data = json!({
+            "trace": { "trace_id": "4BF92F3577B34DA6A3CE929D0E0E4736" }
+        });
+        assert_eq!(
+            extract_trace_id_from_data(&data),
+            Some("4bf92f3577b34da6a3ce929d0e0e4736".to_string())
+        );
+    }
+}