feat(observability): direct-to-cloud OTLP with TLS + auth headers

[taitelee]

Summary

Exports traces/metrics/logs straight to a cloud OTLP gateway (Honeycomb, Grafana Cloud) with no sidecar collector to terminate TLS or inject auth. Closes #97.

The endpoint, TLS, custom/private CA, mutual TLS, and per-RPC auth headers are all configured through the OpenTelemetry SDK's standard OTEL_EXPORTER_OTLP_* environment variables — the same vars every Honeycomb / Grafana Cloud quickstart already documents. InitProvider passes no endpoint/header options and lets the SDK read the env, so WaveHouse owns essentially no config-translation surface of its own.

• Endpoint + scheme-driven TLS — OTEL_EXPORTER_OTLP_ENDPOINT (and the per-signal _TRACES_/_METRICS_/_LOGS_ENDPOINT overrides). An https://URL selects TLS with system root CAs; http:// or a bare host:port stays plaintext. Default localhost:4317.
• Custom CA / mutual TLS — OTEL_EXPORTER_OTLP_CERTIFICATE trusts a private CA; OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE / _CLIENT_KEY enable mTLS.
• Auth headers — OTEL_EXPORTER_OTLP_HEADERS (SDK-parsed, percent-decoded per the OTLP spec) carries cloud auth as gRPC metadata on every signal.

What this removed (vs. the earlier revisions of this PR)

The first cut shipped a WaveHouse-specific translation layer; review steered it toward delegating to the SDK instead. Gone now: the otel.addr/WH_OTEL_ADDR and otel.headers/WH_OTEL_HEADERS keys, the per-signal WH_OTEL_{TRACES,METRICS,LOGS}_ADDR overrides, the custom header parser and its fail-loud boot validation, the TLS 1.3 floor, and endpoint.go (schemeURL/ParseEndpoint). Set OTEL_EXPORTER_OTLP_ENDPOINT in place of WH_OTEL_ADDR. The per-signal enable gates and sample-rate knobs (otel.enabled, otel.traces.sample_rate, …) are unchanged.

Trade-off — fail-soft, by design. Delegating means we inherit the SDK's behavior: a malformed OTEL_EXPORTER_OTLP_HEADERS value is logged via the OTel error handler and skipped, not fatal at boot. We deliberately gave up the "refuse to start with bad auth config" property in exchange for zero parser-parity surface.

The one piece of WaveHouse glue: logExporterTLSOptions

The pinned gRPC logs exporter (otlploggrpc v0.19) reads OTEL_EXPORTER_OTLP_[LOGS_]CERTIFICATE / _CLIENT_CERTIFICATE / _CLIENT_KEY into its config but never wires that *tls.Config into the gRPC dial — its
newGRPCDialOptions only honors the programmatic WithTLSCredentials, otherwise defaulting to system roots. So a custom CA or mTLS supplied via env would silently fall back to system roots for logs only, while traces/metrics
(otlptracegrpc/otlpmetricgrpc v1.43) apply the same env vars correctly. logExporterTLSOptions rebuilds the credentials from those standard env vars and hands them to the log exporter via WithTLSCredentials, so custom-CA + mTLS work
uniformly across all three signals. It returns nil when no custom trust material is set, leaving the public-CA path fully delegated to the SDK. (Confirmed still required as of otlploggrpc v0.20 — a version bump alone doesn't fix it; track upstream before dropping the shim.)

Datadog

Datadog has no public direct-to-cloud OTLP endpoint; its supported path remains the local DDOT Collector embedded in the Datadog Agent, reached as a plaintext receiver. See the deployment guide.

Test plan

Related Issues

Closes #97

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR removes WaveHouse's bespoke OTLP endpoint and header configuration, delegating all endpoint/TLS/authentication setup to the OpenTelemetry SDK's standard environment variables. Exporters are initialized without explicit endpoints; tests, fixtures, and documentation are updated to use OTEL_EXPORTER_OTLP_* env vars, and test utilities gain TLS and gRPC metadata capture.

Changes

SDK-Delegated OTLP Configuration

Layer / File(s)	Summary
Configuration schema removal internal/config/config.go	Removed the OTel.Addr field and dropped the validation that required a non-empty OTLP endpoint when OTel is enabled; updated config comments to point at OTEL_EXPORTER_OTLP_* env vars.
Configuration tests alignment internal/config/config_test.go	Removed default OTel.Addr assertions, stopped populating Addr in sample-rate validation tests, and deleted the test that rejected empty otel.addr when OTel is enabled.
Provider implementation: SDK-driven exporters internal/observability/provider.go	Removed ProviderConfig.Endpoint guidance and changed trace/metric exporter creation to otlptracegrpc.New(ctx) / otlpmetricgrpc.New(ctx) with no explicit endpoint or insecure transport overrides; logs exporter now uses logExporterTLSOptions() and otlploggrpc.New(ctx, ...).
Logs exporter TLS shim internal/observability/logtls.go	Added logExporterTLSOptions() to build TLS credentials from OTEL_EXPORTER_OTLP_* env vars (custom CA, client cert/key) and firstNonEmptyEnv helper for env precedence.
Provider tests: lazy-dial shutdown internal/observability/provider_test.go	Updated shutdown test to remove TCP listener setup and rely on lazy-dial behavior when no OTEL endpoint is configured; removed net import and adjusted comments.
Test infrastructure: TLS-capable FakeOTLP and metadata capture internal/testutil/otlp.go	Added NewFakeOTLPTLS() generating ephemeral self-signed certs, stored server cert PEM, capture incoming gRPC metadata per-signal, added LastTraceHeaders/LastMetricHeaders/LastLogHeaders accessors, and clear captured metadata on Reset.
Integration tests: migrate to env vars tests/integration/otel_test.go	Updated existing tests to set OTEL_EXPORTER_OTLP_ENDPOINT via t.Setenv instead of ProviderConfig.Endpoint and added import for grpc/metadata to assert header propagation.
Integration tests: TLS and header propagation tests/integration/otel_test.go	Added TestOTel_TLSPath_AllSignals (HTTPS endpoint with OTEL_EXPORTER_OTLP_CERTIFICATE) and TestOTel_Headers_AppliedToAllSignals (assert OTEL_EXPORTER_OTLP_HEADERS propagate to traces/metrics/logs).
Startup logging and config examples cmd/wavehouse/main.go, config.yaml	Updated startup observability log to report OTEL_EXPORTER_OTLP_ENDPOINT (with SDK-default fallback) and removed otel.addr from the example config, adding guidance to use OTEL_EXPORTER_OTLP_*.
Documentation: configuration and deployment guides docs/src/content/docs/configuration.md, docs/src/content/docs/deployment.md	Rewrote Observability docs to describe scheme-aware OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, private CA/mTLS vars, local collector / direct-to-cloud / Datadog / Grafana Alloy patterns, and container connectivity examples.
Docs: invariants and changelog AGENTS.md, CHANGELOG.md	Expanded observability invariants to document stdout/OTLP sampling semantics, lazy OTLP dialing, private Prometheus registry, env-var-driven OTLP config, logs TLS shim, and added a changelog bullet for direct-to-cloud OTLP via standard env vars.
E2E fixture update tests/e2e/fixtures/config.yaml	Enabled OTel in the e2e fixture and removed the pinned exporter address to rely on env vars during test runs.

🎯 3 (Moderate) | ⏱️ ~25 minutes

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch otlp-direct-cloud

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch otlp-direct-cloud

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

📚 Docs preview is live → https://cca44c30-wavehouse-docs.wave-rf.workers.dev

• Commit — 71a6b4d: merge main
• Author — @taitelee
• Committed — 2026-06-10 21:11 (UTC-04:00)
• Deployed — 2026-06-10 21:14 EDT

[EricAndrechek]

Hey @taitelee — I went deeper on this one (plus some research into what the OTel Go SDK gives us out of the box) and want to capture it all in one place so we can steer toward as little custom code as possible. Your 0965a08 refactor was the right direction — dropping ParseEndpoint, the per-signal addrs, and the TLS 1.3 floor — this is just continuing that same line of thinking one more step.

TL;DR: Almost all of the endpoint/TLS/header plumbing is something the SDK already does natively via the standard OTEL_* env vars. We can likely delete endpoint.go entirely and shrink the config-translation layer to near-zero.

What the SDK already does for us

otlptracegrpc.New(ctx) (and the metric/log equivalents) with no options automatically read the standard env vars — confirmed against our pinned v1.43.0:

• OTEL_EXPORTER_OTLP_ENDPOINT (+ per-signal _TRACES_/_METRICS_/_LOGS_ENDPOINT) — the scheme drives TLS: https:// ⇒ TLS w/ system roots, http:///bare ⇒ plaintext. This is spec-mandated.
• OTEL_EXPORTER_OTLP_HEADERS (+ per-signal) — SDK-parsed.
• OTEL_EXPORTER_OTLP_CERTIFICATE / _CLIENT_CERTIFICATE / _CLIENT_KEY — custom CA + mTLS, for free.

Refs: otlptracegrpc doc.go, OTLP exporter spec. (go.opentelemetry.io/contrib/exporters/autoexport goes even further and builds all three signals from env, if we ever want gRPC-vs-HTTP selection too.)

Concrete items

1. schemeURL → delete. It only exists because we went through WithEndpointURL, which can't parse a bare host:port. The native WithEndpoint("host:port") + WithTLSCredentials(...)/WithInsecure() takes the bare form directly, and OTEL_EXPORTER_OTLP_ENDPOINT takes a scheme'd URL directly. Either way the shim isn't needed.

2. ParseOTelHeaders has a real conformance bug, not just extra code. The SDK percent-decodes header values (url.PathUnescape inside stringToHeader, in .../otlptracegrpc@v1.43.0/internal/envconfig/envconfig.go); ours stores them raw. So WH_OTEL_HEADERS looks exactly like OTEL_EXPORTER_OTLP_HEADERS but behaves differently. If an operator copies a vendor's value containing a %xx escape (or a percent-encoded comma), we ship the wrong header — silently, with no boot error, which defeats the whole point of the fail-loud parser. Values with a literal , also have no escape hatch in our parser. If we keep any custom parser it needs url.PathUnescape to match the spec — but better is to not have one at all (see Direction).

3. Self-signed / custom-CA gap. Prod TLS is now system-roots-only, and the trust-override hook is -tags integration only — so an operator with a private CA or self-signed gateway can't use the TLS path at all. That's the other half of my original TLS comment: the version-floor concern got fixed, but the trust concern is still open. OTEL_EXPORTER_OTLP_CERTIFICATE solves it for free. Net: the custom path currently costs more code and supports strictly less than delegating to the SDK.

4. PR description is stale. It still advertises per-signal WH_OTEL_{TRACES,METRICS,LOGS}_ADDR, which 0965a08 removed. Worth fixing so the squash message stays accurate.

Direction

Let's lean on the standard OTEL_* env vars as much as we can. Target end state: InitProvider passes no endpoint/header/TLS options and lets the SDK read env; endpoint.go goes away; docs just point operators at the OTel spec vars (which is what every Honeycomb / Grafana Cloud quickstart already tells them anyway). That collapses the new surface area we have to maintain down to basically nothing.

One trade-off to weigh: delegating means we inherit the SDK's fail-soft behavior — a malformed header gets logged via the OTel error handler and skipped, not fatal at boot. We'd lose the "refuse to start with bad auth config" property you built. If you feel strongly about keeping fail-fast, the move is a thin validated pass-through — validate the value at boot, then hand it straight to the SDK with matching (percent-decode) semantics — rather than a separate reimplementation. I lean toward minimal code + accepting fail-soft, but I'm open if you want to keep the boot check. Your call on that one.

None of this touches the provider.go lifecycle work (globals snapshot/rollback, the sync.Once around runtime.Start, the private Prometheus registry) — that part's solid and stays. This is only about the config-translation layer sitting in front of it.

[github-code-quality]

Code Coverage Overview

Languages: Go

Go

The overall coverage in the otlp-direct-cloud branch remains at 89%, unchanged from the main branch.

Show a code coverage summary of the most impacted files.

File	main 46d92e7	otlp-direct-cloud 71a6b4d	+/-
internal/observ...ity/provider.go	80%	79%	-1%
cmd/wavehouse/main.go	68%	68%	0%
internal/config/config.go	94%	94%	0%

---

_{Code Coverage is in Public Preview. Learn more and provide us with your feedback.}