Skip to content

orb(otel): the trace exporter is the one egress redaction-scrub never covers — recordException ships full messages and stacks unscrubbed, to a third-party host picked by an unrelated key #9162

Description

@JSONbored

Problem

src/selfhost/otel.ts is the one egress in the codebase that redaction-scrub is never applied to.

otelSafeAttributes (otel.ts:105-115) re-declares its own local SECRET_KEY regex (:22-23, a copy of
redaction-scrub.ts:13) and applies key-only filtering plus a 160-char truncation. It never imports
scrubString, so SECRET_VALUE, JWT_VALUE, QUERY_SECRET_VALUE, PUBLIC_LOCAL_PATH_SCRUB_PATTERN,
and PRIVATE_TEXT are never applied. A ghs_…, a ?token=… URL, or a /root/… path sitting in a value
under a benign key ("url", "detail", "final") passes verbatim.

Worse, withOtelSpan's catch (otel.ts:331-332) does:

span.recordException(exceptionFor(error));

which writes exception.message and the full exception.stacktrace as span-event attributes —
bypassing otelSafeAttributes entirely, so neither the key filter nor the 160-char cap applies.

Every other sink scrubs values: PostHog via before_send: scrubPostHogEvent, AI errors via
redactSecrets, public comments via sanitizePublicComment. OTel is the exception.

Trigger

An operator sets OTEL_TRACES_EXPORTER=otlp with POSTHOG_API_KEY set and no explicit endpoint.
resolvePostHogOtelTraceTarget (otel.ts:48-53) then silently points the exporter at
https://us.i.posthog.com/i/v1/tracesa third-party host the operator did not name. Any exception
escaping withReviewPipelineSpan (src/queue/processors.ts:3775, 3850, 6749, 10875, 11900, …) or
withReviewSpan("selfhost.ai.provider") (src/selfhost/ai.ts:1352) exports its raw message and stack
there.

Note the coupling: POSTHOG_API_KEY alone is enough to pick the destination; only
OTEL_TRACES_EXPORTER gates whether anything is sent. An operator who set the key for product analytics
may not realise they also selected a trace destination.

Impact

A redaction boundary the codebase treats as universal has a hole at exactly the egress that leaves the
operator's network by default.

Requirements

  • Import scrubString from redaction-scrub into otelSafeAttributes, deleting the duplicated local
    SECRET_KEY so there is one definition.
  • Replace span.recordException(err) with a scrubbed, length-capped
    span.addEvent("exception", otelSafeAttributes({...})).
  • Require an explicit opt-in (or at least an explicit endpoint) before defaulting the trace destination to
    a third-party host derived from an unrelated key.
  • Add an invariant test that every span attribute and span event passes through the shared scrubber.

Test Coverage Requirements

99%+ patch coverage, branch-counted; a token-shaped value under a benign key must be scrubbed in both
attribute and exception-event paths.

Links & Resources

maintainer-only — confidentiality boundary.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions