Skip to content

ai(observability): PostHog's conversation, sentiment and evaluation views are empty — $ai_input/$ai_output_choices are never populated #10218

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability).

Problem

PostHog's LLM Analytics Traces view is almost entirely empty for this project. Every row shows for Input message, Output message, Sentiment, Tools, Token usage and Review; only latency and cost are populated.

$ai_input and $ai_output_choices are deliberately never set (src/selfhost/posthog.ts), and a test asserts their absence. That was a reasonable default — but it also means the whole content-dependent half of the product is unavailable: the conversation view, sentiment, evaluations, LLM-as-judge, and online evals all read the captured prompt and completion.

Token usage is a separate root cause, tracked in #10211.

What this is asking for

Make content capture possible, without changing what a self-hoster gets by default.

  • An explicit env opt-in, default OFF, so upgrading an ORB never starts shipping private PR diffs and model completions to a vendor as a side effect of a version bump.
  • Content routed through the existing before_send redaction (scrubRecord / scrubString) rather than a second, weaker path.
  • A size cap. A review prompt carries up to 120k chars of diff plus a 240k-char aggregate context budget; an uncapped capture would exceed PostHog's payload limits and be dropped wholesale.
  • Images dropped rather than base64-encoded into a telemetry event.
  • The self-hosting doc updated in the same change — it currently promises metadata-only unconditionally and would become false.
  • The existing absence-assertion test flipped to cover both states rather than deleted.

Deliverables

  • LOOPOVER_POSTHOG_AI_CONTENT (truthy-string, default off) gates $ai_input / $ai_output_choices.
  • LOOPOVER_POSTHOG_AI_CONTENT_MAX_CHARS caps each message; a non-numeric, zero or negative override falls back to the default rather than disabling the cap.
  • Truncated content is marked as truncated, so a clipped prompt is never mistaken for the whole one.
  • The gate lives in one place (the capture function), not at each call site, so a future call site cannot forget it.
  • Embedding requests report their raw texts; chat requests report the flattened messages.
  • The failure path captures the prompt (a stall or a context-length rejection is only diagnosable against its input) and no completion.
  • Docs updated with the privacy consequences, not just the var name.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions