Part of epic #8286 (Phase 3 — AI observability).
Problem
Two independent faults make PostHog's AI observability structurally unable to answer the two questions that matter most for a product where AI is the product.
1. Every generation is its own trace
capturePostHogAiGeneration (src/selfhost/posthog.ts) sets $ai_trace_id: randomUUID() on every call. Live project state: 13,428 AI events across 13,428 distinct trace IDs. A review fans out across RAG embeddings, two dual-review legs, N retries per model, and any self-consistency runs — and every one of those lands as an unrelated single-event trace. The trace view is unusable, and "what did this review actually do, and where did the time and money go" is unanswerable.
The fix is already sitting in the file. operationalProperties calls currentOtelTraceIds() and attaches the real ids as plain trace_id / span_id properties — then the line below mints a fresh UUID and PostHog groups by that instead. withReviewPipelineSpan (src/selfhost/review-tracing.ts) already wraps a whole review, so the ambient trace id is exactly the grouping key wanted.
$ai_span_id, $ai_parent_id, and $ai_span_name are never set at all, so even within a trace there is no tree.
2. AI spend is not attributable
Every AI event carries distinct_id: "loopover-selfhost" and no $groups. There is no per-repo, per-installation, or per-PR dimension in PostHog. "Which repo costs the most in AI spend" is answerable only by SQL against ai_usage_events on the ORB — not in PostHog, not for the miner, and not at all for embeddings/vision/advisory calls, which carry no repo even as a flat property.
For scale: claude-sonnet-5 alone is $489.87 across 2,282 calls in ~5 days.
Deliverables
Boundaries
Content capture ($ai_input / $ai_output_choices) is deliberately out of scope — the metadata-only policy is documented in self-hosting-operations.mdx and changing it is a privacy decision, not a wiring one. Instrumenting the Worker/BYOK paths, the miner's zeroed token counts, and the eval signals are their own issues.
Part of epic #8286 (Phase 3 — AI observability).
Problem
Two independent faults make PostHog's AI observability structurally unable to answer the two questions that matter most for a product where AI is the product.
1. Every generation is its own trace
capturePostHogAiGeneration(src/selfhost/posthog.ts) sets$ai_trace_id: randomUUID()on every call. Live project state: 13,428 AI events across 13,428 distinct trace IDs. A review fans out across RAG embeddings, two dual-review legs, N retries per model, and any self-consistency runs — and every one of those lands as an unrelated single-event trace. The trace view is unusable, and "what did this review actually do, and where did the time and money go" is unanswerable.The fix is already sitting in the file.
operationalPropertiescallscurrentOtelTraceIds()and attaches the real ids as plaintrace_id/span_idproperties — then the line below mints a fresh UUID and PostHog groups by that instead.withReviewPipelineSpan(src/selfhost/review-tracing.ts) already wraps a whole review, so the ambient trace id is exactly the grouping key wanted.$ai_span_id,$ai_parent_id, and$ai_span_nameare never set at all, so even within a trace there is no tree.2. AI spend is not attributable
Every AI event carries
distinct_id: "loopover-selfhost"and no$groups. There is no per-repo, per-installation, or per-PR dimension in PostHog. "Which repo costs the most in AI spend" is answerable only by SQL againstai_usage_eventson the ORB — not in PostHog, not for the miner, and not at all for embeddings/vision/advisory calls, which carry no repo even as a flat property.For scale:
claude-sonnet-5alone is $489.87 across 2,282 calls in ~5 days.Deliverables
$ai_trace_iduses the ambient OTel trace id when there is one, falling back to a minted UUID only when there genuinely is no span (AI_EMBED/AI_VISION/AI_ADVISORYrun outside review spans — an orphan trace is valid, it just must not claim a span id it does not have).$ai_span_idfrom the ambient span, and$ai_span_nameso the trace tree has readable nodes.repogroup on every AI event. It must read the ALREADY-PROCESSED repo value, so that under the sharedLOOPOVER_CENTRAL_POSTHOG_KEYthe group inherits the existing HMAC anonymization and its fail-closed behavior — a raw private repo name must never reach the group index.Boundaries
Content capture (
$ai_input/$ai_output_choices) is deliberately out of scope — the metadata-only policy is documented inself-hosting-operations.mdxand changing it is a privacy decision, not a wiring one. Instrumenting the Worker/BYOK paths, the miner's zeroed token counts, and the eval signals are their own issues.