Skip to content

ai(observability): every generation is its own trace, and AI spend has no repo attribution #10185

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability).

Problem

Two independent faults make PostHog's AI observability structurally unable to answer the two questions that matter most for a product where AI is the product.

1. Every generation is its own trace

capturePostHogAiGeneration (src/selfhost/posthog.ts) sets $ai_trace_id: randomUUID() on every call. Live project state: 13,428 AI events across 13,428 distinct trace IDs. A review fans out across RAG embeddings, two dual-review legs, N retries per model, and any self-consistency runs — and every one of those lands as an unrelated single-event trace. The trace view is unusable, and "what did this review actually do, and where did the time and money go" is unanswerable.

The fix is already sitting in the file. operationalProperties calls currentOtelTraceIds() and attaches the real ids as plain trace_id / span_id properties — then the line below mints a fresh UUID and PostHog groups by that instead. withReviewPipelineSpan (src/selfhost/review-tracing.ts) already wraps a whole review, so the ambient trace id is exactly the grouping key wanted.

$ai_span_id, $ai_parent_id, and $ai_span_name are never set at all, so even within a trace there is no tree.

2. AI spend is not attributable

Every AI event carries distinct_id: "loopover-selfhost" and no $groups. There is no per-repo, per-installation, or per-PR dimension in PostHog. "Which repo costs the most in AI spend" is answerable only by SQL against ai_usage_events on the ORB — not in PostHog, not for the miner, and not at all for embeddings/vision/advisory calls, which carry no repo even as a flat property.

For scale: claude-sonnet-5 alone is $489.87 across 2,282 calls in ~5 days.

Deliverables

  • $ai_trace_id uses the ambient OTel trace id when there is one, falling back to a minted UUID only when there genuinely is no span (AI_EMBED / AI_VISION / AI_ADVISORY run outside review spans — an orphan trace is valid, it just must not claim a span id it does not have).
  • Set $ai_span_id from the ambient span, and $ai_span_name so the trace tree has readable nodes.
  • Stamp a repo group on every AI event. It must read the ALREADY-PROCESSED repo value, so that under the shared LOOPOVER_CENTRAL_POSTHOG_KEY the group inherits the existing HMAC anonymization and its fail-closed behavior — a raw private repo name must never reach the group index.
  • Both branches of each tested (ambient span vs none, repo present vs dropped).

Boundaries

Content capture ($ai_input / $ai_output_choices) is deliberately out of scope — the metadata-only policy is documented in self-hosting-operations.mdx and changing it is a privacy decision, not a wiring one. Instrumenting the Worker/BYOK paths, the miner's zeroed token counts, and the eval signals are their own issues.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions