feat(observability): direct-to-cloud OTLP (#97)

AGENTS.md

@@ -42,7 +42,7 @@ Twelve internal packages under `internal/`:
 12. **Structured queries**: Type-safe query AST endpoint (`POST /v1/tables/{table}/query`) validated against schema, with permission enforcement, timestamp bucketing for cache optimization, and `DefaultMaxRows` (10,000) limit cap.
 13. **Named query pipes**: Pre-defined SQL templates (inspired by Tinybird) with parameter binding, role restrictions, and caching. Stored in NATS KV with `.sql` file directory bootstrap.
 14. **TypeScript SDK**: `@wavehouse/sdk` — zero-dependency client with typed query builder, real-time SSE, live queries with smart aggregation classification (incrementable/decomposable/poll), and codegen CLI.
-15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. When changing the logger, the sampler, or the provider wiring, preserve these invariants.
+15. **Observability invariants**: Stdout is *always* 100% — the slog logger fans out to stdout AND OTLP, and stdout sampling would silently hide records that scraping pipelines (Promtail/Alloy/Vector → Loki) are paying to store. Sampling knobs apply only to OTLP push. WARN+ERROR records always export at 100% regardless of `logs.sample_rate` — silently dropping errors during incidents would be a worse failure mode than the cost of forwarding them all (this is a *non-configurable* floor; do not expose it). gRPC OTel exporters dial lazily, so an unreachable collector never blocks startup; transient failures surface via the OTel SDK's error handler. The OTel Prometheus exporter (when enabled) uses a *private* `prometheus.Registry` to avoid leaking process/Go collectors that `prometheus.DefaultRegisterer` auto-registers into our `/metrics` output. TLS is selected via *scheme sniffing* on `otel.addr` (`https://` → TLS, anything else → plaintext) — there is no separate `otel.tls.enabled` boolean. `otel.headers` is applied uniformly to all OTLP exporters (no per-signal header override); per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) are the supported way to split signals across gateways. When changing the logger, the sampler, or the provider wiring, preserve these invariants.
 16. **Bearer-token-only CORS posture**: WaveHouse is a Bearer-token API — `Authorization: Bearer <jwt>` on every authenticated request, no cookies, no session middleware. The CORS middleware (`internal/api/router.go` `corsMiddleware`) deliberately **never** emits `Access-Control-Allow-Credentials`, because (a) we don't need it (Bearer tokens are explicit request headers, not browser-managed credentials) and (b) the historical pairing of `Allow-Credentials: true` with `Allow-Origin: *` is a CORS spec violation that browsers reject. The `cors_allowed_origins` allowlist controls *which origins can read responses*, not cookie scope. CSRF protection is structural: cross-site requests can't smuggle a Bearer token because the browser won't auto-attach `Authorization` headers cross-origin. Do not reintroduce cookie-based auth or `Access-Control-Allow-Credentials` without a separate design discussion — the current posture is the answer to GitHub issues #29 and #30.
 17. **Non-fatal boot**: Schema-discovery failure on boot (ClickHouse unreachable, missing database, transient network blip) is non-fatal — `cmd/wavehouse` records an `api.BootState` diagnostic, binds `:8080`, and retries via `SchemaRegistry.RetryRefresh` (exp backoff 2s → 60s) in the background. `/health` and `/ready` return 503 with the latest diagnostic until a Refresh succeeds, after which `/health` stays 200 for the rest of the process. Keeps supervisor restart loops bounded and gives operators a queryable failure surface (`curl /health`) instead of a restart-log grep.
 

CHANGELOG.md

@@ -36,6 +36,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Dependabot auto-merge no longer slipped past CI on consolidated-pipeline PRs** (`main branch protection` ruleset; `.github/workflows/project-orchestrator.yml`; `.github/workflows/claude-review.yml`; `AGENTS.md`): on commit `93b3206` the previously six-job CI was collapsed into a single job named `CI`, but three places still referenced the old multi-job names — the ruleset's `required_status_checks` (`Validate` + `Admin approval` only — `CI` not listed), the `bot-clean` check in `project-orchestrator.yml` (looking for `Build`/`Validate`/`Lint`/`Test`/`Integration Tests`/`SDK Tests`), and `REQUIRED_CHECKS` in `claude-review.yml` (same six). Net effect: a Dependabot PR opened at T+0 would have its `Validate` and `Admin approval` checks satisfied within ~30s (PR-title workflow + the auto-approval workflow), `gh pr merge --auto` would fire, GitHub would see all *required* checks green, and the PR would squash-merge before `ci.yml` even started running. Pre-consolidation runs (PR #90 etc.) didn't expose this because GitHub's auto-merge waits for in-flight checks as a courtesy, and the multi-job CI was producing checks before auto-merge fired — but with one collapsed `CI` check that didn't start until later, the courtesy wait disappeared. Fix: added `CI` to the ruleset's required-checks (now `Admin approval` / `CI` / `Validate`) via the `gh api PUT /repos/Wave-RF/WaveHouse/rulesets/15353356` round-trip, and updated both bot-clean lists to look for `CI` + `Validate`. AGENTS.md note about the required-check set updated to match. Diagnosis steps recorded inline as code comments in both workflow files so future-me knows where to look if the check name changes again.
 
 ### Added
+- **Direct-to-cloud OTLP — TLS + auth headers, no sidecar required** (`internal/config/config.go`, `internal/observability/provider.go`, `internal/observability/endpoint.go`, `internal/testutil/otlp.go`, `cmd/wavehouse/main.go`, `tests/integration/otel_test.go`, `docs/src/content/docs/{configuration,deployment}.md`, `AGENTS.md`): closes [#97](https://github.com/Wave-RF/WaveHouse/issues/97). `otel.addr` is now scheme-aware — `https://` selects TLS (system root CAs by default), `http://` or a bare `host:port` stays plaintext gRPC (backward-compat with the prior `WithInsecure()` default). New `otel.headers` (`WH_OTEL_HEADERS`) takes the OTel spec env-var format — comma-separated `key=value`, values may contain `=` so base64 padding round-trips — and applies the parsed map as gRPC metadata to every OTLP exporter (traces, metrics, logs). Together these unlock direct push to Honeycomb and Grafana Cloud's OTLP gateway without a sidecar. Datadog does not publish a direct-to-cloud OTLP endpoint — its supported path remains the DDOT Collector embedded in the Datadog Agent (host or k8s), which WaveHouse reaches as a plaintext local OTLP receiver on `127.0.0.1:4317` (no headers — the API-key auth lives on the Agent); see `docs/deployment.md` for the worked example. Per-signal endpoint overrides (`otel.{traces,metrics,logs}.addr`) point one signal at a different gateway than the top-level default (Grafana Cloud uses distinct trace/metric/log hosts in some regions); empty means inherit. Headers always apply uniformly — there is intentionally no per-signal header override. Test surface: `FakeOTLP` gains a `NewFakeOTLPTLS(t)` constructor (ephemeral self-signed cert + matching client `*tls.Config`) and per-signal `LastTraceHeaders / LastMetricHeaders / LastLogHeaders` accessors via captured gRPC metadata. New integration tests pin the TLS dial path, headers-on-all-signals propagation, and per-signal endpoint splitting. Validation rejects malformed `otel.headers` (missing `=`, empty key) at boot rather than letting a typo silently disable auth in production. mTLS / client-certificate auth is not yet supported.
 - **Astro / Starlight documentation site at `wavehouse.dev`** (`docs/`, `Makefile`, `.github/dependabot.yml`, `.github/workflows/ci.yml`, `.gitignore`, `.vscode/launch.json`): full content + tooling for the public docs (Getting Started, Why WaveHouse, Architecture, API Reference, TypeScript SDK, Configuration, Deployment, Development). Build-time mermaid via `rehype-mermaid` (inline-svg strategy, no client-side JS), LaTeX via `remark-math` + `rehype-katex`, image zoom via `starlight-image-zoom`, expressiveCode with github dark/light + `env`/`dns` Shiki language aliases. PostHog frontend telemetry (key is public by design — embedded in every visitor's browser). `editLink` pointed at `main`; `lastUpdated` on. Sidebar lives in `docs/src/config/sidebar.ts` as the single source of truth, consumed by both Starlight rendering and the LLM-friendly outputs below. Cloudflare Workers + Static Assets deployment via `docs/wrangler.jsonc` (no auto-deploy yet — wired up when the GH Actions workflow lands separately). Make targets stay minimal: `dev-docs` (Astro dev server on :4321), `build-docs`, `preview-docs` — the last serves the production build through `wrangler dev`, so the Cloudflare Worker's content negotiation (`Accept: text/markdown` / LLM-bot User-Agent → `.md` twin) is exercised end-to-end the same way it will be in production.
 - **Per-page `.md` twin + `llms.txt` family via `starlight-llm-tools`** (`docs/astro.config.mjs`, `docs/worker/index.ts`, `docs/wrangler.jsonc`): every doc is also served as raw markdown at `<path>.md` with a navigation header (Section / Subpages / Related / HTML version pointers), plus three concatenated views — `/llms.txt` manifest, `/llms-full.txt` (every page, sidebar-ordered), `/llms-small.txt` (overview pages only). Page-title slot gains a **Copy Markdown** button and **Open with AI** dropdown (Claude / ChatGPT / Cursor). Content negotiation is handled by a Cloudflare Worker that re-exports the `cloudflare-md-router` package's default handler in one line — `Accept: text/markdown` or known LLM-bot User-Agent (GPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, etc.) gets the `.md` twin transparently, with a fall-through to the HTML response when the twin doesn't exist.
 - **Two extracted plugin packages, MIT-licensed**: [`github.com/Wave-RF/cloudflare-md-router`](https://github.com/Wave-RF/cloudflare-md-router) (Workers handler + `createMdRouter()` factory + extensible bot-UA regex; consumed in `docs/worker/index.ts` as a one-line re-export) and [`github.com/Wave-RF/starlight-llm-tools`](https://github.com/Wave-RF/starlight-llm-tools) (Starlight plugin that auto-injects the four routes, the two components, and the `PageTitle` override; optionally calls into `starlight-glossary/transform` when present). Both pinned via `github:Wave-RF/...` in `docs/package.json`; lockfile pins the resolved commit SHAs. Pre-npm-publish state — switch to normal version specifiers when the packages land on npm.

cmd/wavehouse/main.go

@@ -98,26 +98,41 @@ func run() int {
 	slog.SetDefault(logger)
 
 	var promHandler http.Handler
-	// Provider init runs whenever either OTLP push or Prometheus exposition is
-	// wanted — Prometheus-only operation (Alloy/scrape, no collector) is a
-	// first-class mode. The OTel SDK MeterProvider is the shared substrate.
 	if cfg.OTel.Enabled || cfg.Prometheus.Enabled {
+		// Validate() already ran validateOTelHeaders; this re-parse exists
+		// only to catch drift between the two parsers (header_parity_test.go
+		// pins them at CI time, but a runtime sanity check is cheap). Skip
+		// in Prometheus-only mode — Validate() skips headers there too, so
+		// a stale WH_OTEL_HEADERS would produce a misleading error.
+		var headers map[string]string
+		if cfg.OTel.Enabled {
+			var err error
+			headers, err = observability.ParseOTelHeaders(cfg.OTel.Headers)
+			if err != nil {
+				logger.Error("otel.headers parse failed after Validate accepted them; refusing to start with bad auth config", "error", err)
+				return 1
+			}
+		}
 		otelShutdown, ph, err := observability.InitProvider(ctx, serviceName, observability.ProviderConfig{
 			Endpoint:          cfg.OTel.Addr,
+			Headers:           headers,
+			TracesEndpoint:    cfg.OTel.Traces.Addr,
 			TracesEnabled:     cfg.OTel.Enabled && cfg.OTel.Traces.Enabled,
 			TracesSampleRate:  cfg.OTel.Traces.SampleRate,
+			MetricsEndpoint:   cfg.OTel.Metrics.Addr,
 			MetricsEnabled:    cfg.OTel.Enabled && cfg.OTel.Metrics.Enabled,
 			PrometheusEnabled: cfg.Prometheus.Enabled,
+			LogsEndpoint:      cfg.OTel.Logs.Addr,
 			LogsEnabled:       cfg.OTel.Enabled && cfg.OTel.Logs.Enabled,
 		})
 		if err != nil {
 			logger.Error("failed to initialize observability, falling back to stdout", "error", err)
 		} else {
 			promHandler = ph
 			defer func() {
-				// Bound shutdown so an unreachable collector doesn't hang
-				// process exit. The OTel SDK's batch processors don't fully
-				// honor the context deadline during gRPC retry/backoff.
+				// Bounded — OTel batch processors don't fully honor context
+				// deadlines during gRPC retry/backoff against an unreachable
+				// collector.
 				ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 				defer cancel()
 				_ = otelShutdown(ctx)

config.yaml

@@ -19,14 +19,18 @@ server:
 
 otel:
   enabled: false         # master switch — set true to export via OTLP gRPC
-  addr: "127.0.0.1:4317"
+  addr: "127.0.0.1:4317" # `https://...` → TLS, `http://...` or bare host:port → plaintext
+  # headers: ""          # comma-separated key=value, e.g. "x-honeycomb-team=API_KEY"
   traces:
     enabled: true
+    # addr: ""           # per-signal endpoint override; empty inherits otel.addr
     sample_rate: 1.0     # head-based, [0.0, 1.0]; tune down for high QPS
   metrics:
     enabled: true        # OTLP push for metrics
+    # addr: ""           # per-signal endpoint override; empty inherits otel.addr
   logs:
     enabled: true
+    # addr: ""           # per-signal endpoint override; empty inherits otel.addr
     sample_rate: 1.0     # DEBUG/INFO OTLP rate; WARN+ always 100%, stdout always 100%
 
 # Prometheus exposition is independent of OTel — works on its own (Alloy /

deployments/compose/standalone.yaml

@@ -32,6 +32,8 @@ services:
       # read-only and uncomment WH_PIPES_DIR. The directory is a seed for
       # NATS KV — runtime pipe edits go through the API, not the files.
       # WH_PIPES_DIR: /app/pipes
+      # Observability env vars (WH_OTEL_*, WH_PROMETHEUS_*) are documented in
+      # docs/src/content/docs/deployment.md → "Observability via Compose".
     volumes:
       - wavehouse-data:/app/data
       # - ./my-pipes:/app/pipes:ro    # uncomment alongside WH_PIPES_DIR

docs/src/content/docs/configuration.md

@@ -113,16 +113,22 @@ The master switch is `otel.enabled`. When `true`, each signal (traces/metrics/lo
 
 **Sampling rates apply only to the OTLP push path.** Stdout always emits 100% of records — operators using a scraping-style pipeline (Promtail/Grafana Alloy → Loki, Vector, Fluent Bit, etc.) set the collection rate at the scraper, not the application. WaveHouse pushes telemetry to an OTel collector; the scraper world owns its own ingest policy. If you want to throttle OTLP volume for cost, lower the rates below. If you want to throttle Loki/Datadog Logs/etc., do it at that pipeline.
 
-**TLS / direct-to-cloud limitation.** The OTLP exporters currently use `WithInsecure()` — plaintext gRPC only. WaveHouse cannot ship directly to TLS-protected OTLP endpoints (Grafana Cloud's OTLP gateway, Honeycomb, Datadog OTLP, etc.). The standard workaround is a sidecar collector (the OTel collector or Grafana Alloy) on `127.0.0.1:4317` that receives our plaintext OTLP and re-exports to the cloud endpoint with TLS + auth headers configured locally. Tracked in #97.
+**TLS + direct-to-cloud OTLP.** `otel.addr` is scheme-aware: `https://` selects TLS (system root CAs), `http://` or a bare `host:port` stays plaintext gRPC. Combined with `otel.headers` for auth, WaveHouse can ship telemetry straight to TLS-protected cloud endpoints (Grafana Cloud's OTLP gateway, Honeycomb, etc.) without a sidecar collector. Datadog has no public direct-to-cloud OTLP endpoint — use the local DDOT Collector path described in the [deployment guide](./deployment.md#pattern-datadog-via-local-ddot-collector) instead. A sidecar is still useful for egress queuing, batching, and tail-based sampling — it's just no longer required. See [Deployment → Direct-to-cloud OTLP](./deployment.md#pattern-direct-to-cloud-otlp-honeycomb-grafana-cloud) for worked examples.
+
+If different signals need different gateway hosts (Grafana Cloud's distinct trace / metric / log endpoints, say), set `otel.{traces,metrics,logs}.addr` to override the default for that signal — empty means inherit from `otel.addr`.
 
 | YAML Key | Env Var | Default | Description |
 | -------- | ------- | ------- | ----------- |
 | `otel.enabled` | `WH_OTEL_ENABLED` | `false` | Master switch. When `false`, no signals are initialized regardless of the sub-toggles below. |
-| `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | OTLP gRPC endpoint used by every enabled signal. Plain `host:port` — no scheme, plaintext gRPC only (see TLS note above). See `deployments/signoz/` for a local collector setup. |
+| `otel.addr` | `WH_OTEL_ADDR` | `127.0.0.1:4317` | Default OTLP gRPC endpoint. Accepts `host:port` (plaintext), `http://host:port` (plaintext), or `https://host:port` (TLS via system root CAs). A trailing URL path is tolerated and stripped — gRPC routes by service name. See `deployments/signoz/` for a local plaintext collector setup. |
+| `otel.headers` | `WH_OTEL_HEADERS` | *(empty)* | Comma-separated `key=value` pairs applied as gRPC metadata to every OTLP export — the standard auth knob for cloud endpoints (`authorization=Basic <b64>`, `x-honeycomb-team=<key>`). Values may contain `=`; only the first `=` per segment splits key from value (base64 padding round-trips). Validated at config load. |
 | `otel.traces.enabled` | `WH_OTEL_TRACES_ENABLED` | `true` | Export traces via OTLP gRPC. |
+| `otel.traces.addr` | `WH_OTEL_TRACES_ADDR` | *(empty)* | Per-signal override for `otel.addr` (same scheme rules). Empty means inherit. |
 | `otel.traces.sample_rate` | `WH_OTEL_TRACES_SAMPLE_RATE` | `1.0` | Head-based trace sampling rate in `[0.0, 1.0]`. `1.0` exports every trace; `0.0` exports none. Defaults to 100% (matches the OpenTelemetry SDK default); lower it for high-QPS production services where collector or backend cost is a concern. Best practice is "100% at the source, downsample at the collector" via tail-based sampling. Validated at config load. |
 | `otel.metrics.enabled` | `WH_OTEL_METRICS_ENABLED` | `true` | Export metrics + Go runtime metrics via OTLP gRPC. Periodic reader interval is fixed at 15s. Metrics are pre-aggregated so there is no sampling knob. |
+| `otel.metrics.addr` | `WH_OTEL_METRICS_ADDR` | *(empty)* | Per-signal override for `otel.addr`. Empty means inherit. |
 | `otel.logs.enabled` | `WH_OTEL_LOGS_ENABLED` | `true` | Export logs via OTLP gRPC. Disabling this leaves stdout logging untouched — the OTel logger provider is simply not registered. |
+| `otel.logs.addr` | `WH_OTEL_LOGS_ADDR` | *(empty)* | Per-signal override for `otel.addr`. Empty means inherit. |
 | `otel.logs.sample_rate` | `WH_OTEL_LOGS_SAMPLE_RATE` | `1.0` | OTLP export rate for `DEBUG`/`INFO` records, in `[0.0, 1.0]`. Validated at config load. `WARN` and `ERROR` records always export at 100% — dropping them silently during incidents is too dangerous to expose as a knob. **Stdout always receives 100% of records regardless of this rate** (see the scraper note above). |
 
 ### Prometheus