Skip to content

ai(observability): runChatGrounding and the runCodingAgentAttempt driver path emit no $ai_* telemetry #10200

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability). Follow-up to #10198.

Problem

Two AMS Agent-SDK surfaces make real model calls and emit no $ai_* telemetry at all, so their spend, latency and failure rate are invisible in PostHog.

1. runChatGrounding — never instrumented

packages/loopover-engine/src/miner/chat-grounding.ts drives a full query() session (up to CHAT_MAX_TURNS) with MCP tools attached, and folds the stream into wire events. Nothing on that path captures a generation. A failure becomes a chat_grounding_failed wire event and a provider-config problem becomes an error event — neither reaches PostHog.

Note this driver does not currently surface a usage/result frame at all, so instrumenting it also means folding the SDK's result message (its usage and total_cost_usd) rather than only the assistant/user messages it reads today.

2. runCodingAgentAttempt bypasses the capture wrapper

withCodingAgentAiGenerationCapture is applied inside constructProductionCodingAgentDriver (packages/loopover-miner/lib/coding-agent-construction.ts). The runCodingAgentAttempt path calls createCodingAgentDriver directly (packages/loopover-engine/src/miner/driver-factory.ts), so it never passes through that construction site and every attempt it runs is uncaptured.

That placement is why the bypass is silent: the wrapper is attached at one of two construction sites, and the other one is a plain factory call that looks complete on its own.

Deliverables

  • Instrument runChatGrounding with an $ai_generation capture carrying real model/provider/latency, and tokens + cost from the SDK's result frame.
  • Ensure every coding-agent attempt is captured regardless of which construction path built the driver — preferably so that a future third call site cannot silently opt out again, rather than by adding the wrapper at a second site.
  • Cover the failure paths, not just the happy path: a chat-grounding provider-config error and an SDK throw should both be visible.
  • Keep the metadata-only policy — no prompt, transcript or tool output on any event.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions