Part of epic #8286 (Phase 3 — AI observability).
Problem
PostHog's LLM Analytics Traces view is almost entirely empty for this project. Every row shows – for Input message, Output message, Sentiment, Tools, Token usage and Review; only latency and cost are populated.
$ai_input and $ai_output_choices are deliberately never set (src/selfhost/posthog.ts), and a test asserts their absence. That was a reasonable default — but it also means the whole content-dependent half of the product is unavailable: the conversation view, sentiment, evaluations, LLM-as-judge, and online evals all read the captured prompt and completion.
Token usage is a separate root cause, tracked in #10211.
What this is asking for
Make content capture possible, without changing what a self-hoster gets by default.
- An explicit env opt-in, default OFF, so upgrading an ORB never starts shipping private PR diffs and model completions to a vendor as a side effect of a version bump.
- Content routed through the existing
before_send redaction (scrubRecord / scrubString) rather than a second, weaker path.
- A size cap. A review prompt carries up to 120k chars of diff plus a 240k-char aggregate context budget; an uncapped capture would exceed PostHog's payload limits and be dropped wholesale.
- Images dropped rather than base64-encoded into a telemetry event.
- The self-hosting doc updated in the same change — it currently promises metadata-only unconditionally and would become false.
- The existing absence-assertion test flipped to cover both states rather than deleted.
Deliverables
Part of epic #8286 (Phase 3 — AI observability).
Problem
PostHog's LLM Analytics Traces view is almost entirely empty for this project. Every row shows
–for Input message, Output message, Sentiment, Tools, Token usage and Review; only latency and cost are populated.$ai_inputand$ai_output_choicesare deliberately never set (src/selfhost/posthog.ts), and a test asserts their absence. That was a reasonable default — but it also means the whole content-dependent half of the product is unavailable: the conversation view, sentiment, evaluations, LLM-as-judge, and online evals all read the captured prompt and completion.Token usage is a separate root cause, tracked in #10211.
What this is asking for
Make content capture possible, without changing what a self-hoster gets by default.
before_sendredaction (scrubRecord/scrubString) rather than a second, weaker path.Deliverables
LOOPOVER_POSTHOG_AI_CONTENT(truthy-string, default off) gates$ai_input/$ai_output_choices.LOOPOVER_POSTHOG_AI_CONTENT_MAX_CHARScaps each message; a non-numeric, zero or negative override falls back to the default rather than disabling the cap.