Skip to content

ai(observability): the miner reports 0 input/output tokens because both drivers sum the split away #10198

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability).

Problem

The miner's $ai_generation events report 0 input tokens and 0 output tokens for every AMS coding-agent attempt, so PostHog's own LLM cost views are blind to the miner entirely. The real figure is emitted, but under a non-standard tokens_used property that those views do not read.

captureMinerPostHogAiGeneration (packages/loopover-miner/lib/posthog.ts) hardcodes them:

$ai_input_tokens: 0,
$ai_output_tokens: 0,
...
if (Number.isFinite(event.totalTokens)) properties.tokens_used = event.totalTokens;

Its doc comment explains this as honest: "there is no input/output split available at this layer, so this deliberately does NOT populate $ai_input_tokens/$ai_output_tokens with a fabricated split".

The split does exist — it is discarded one layer down

That reasoning is correct about the layer, but the layer below it throws the split away. Both engine drivers read the two sides separately and then return only their sum:

packages/loopover-engine/src/miner/agent-sdk-driver.ts:

function tokensFromResultMessage(resultMessage) {
  const inputTokens = finiteNonNegativeNumber(usage?.input_tokens);
  const outputTokens = finiteNonNegativeNumber(usage?.output_tokens);
  if (inputTokens === undefined && outputTokens === undefined) return undefined;
  return (inputTokens ?? 0) + (outputTokens ?? 0);   // <- the split is dropped here
}

packages/loopover-engine/src/miner/cli-subprocess-driver.ts's totalTokensFromUsage does the same with usage.inputTokens/usage.outputTokens.

So CodingAgentDriverResult carries only a blended tokensUsed, and the miner is left with nothing real to report.

Deliverables

  • Carry inputTokens/outputTokens on CodingAgentDriverResult alongside the existing blended tokensUsed, populated by both drivers from the values they already read.
  • Populate $ai_input_tokens/$ai_output_tokens from the real split so PostHog's cost views see miner spend.
  • Never fabricate one: a provider that reports only a blended total (a CLI that emits total_tokens and nothing else) must leave the split absent, and the blended figure keeps riding in tokens_used.
  • A side the provider did not report, or reported out-of-contract (negative/NaN/Infinity), stays absent rather than becoming a 0 that is indistinguishable from a real 0 once aggregated.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions