Pi Coding Agent instrumentation double-counts tokens and estimated cost in trace rollups (AgentSession.prompt carries summed LLM usage) #2659

Summary

With wrapPiCodingAgentSDK (braintrust 3.39.0, @earendil-works/pi-coding-agent 1.4.0), each AgentSession.prompt task span gets the summed token usage of its child llm spans, and it also carries metadata.model. Braintrust's trace rollups add the parent's metrics to the children's. As a result, the trace summary, the logs table and the trace header report exactly 2× the prompt, completion and cached tokens and 2× estimated_cost for every Pi run.

The instrumentation spec says parent task spans "MAY include aggregate token metrics for display, but SDKs SHOULD avoid making parent rollups look like independent billable LLM calls" (braintrust-spec/skills/instrumentation-spec/references/features/token-and-cost-metrics.md). Here the parent is priced and summed like an LLM call.

Where

js/src/instrumentation/plugins/pi-coding-agent-plugin.ts at braintrust@3.39.0:

• finishPiLlmSpan logs the usage on the LLM span and also calls addMetrics(promptState.metrics, usageMetrics) (L703-L707).
• finishPiPromptRun logs cleanMetrics(state.metrics) on the AgentSession.prompt span, together with extractModelMetadata(...) (L655-L667).

Minimal reproduction

1. Call braintrust.wrapPiCodingAgentSDK(pi), create an AgentSession with any model, and run one session.prompt(...) that takes at least one LLM turn under initLogger(...).
2. Run these two project-log queries on that trace:

-- trace rollup
SELECT id, metrics.prompt_tokens, metrics.estimated_cost
FROM project_logs('<project_id>', shape => 'summary') WHERE root_span_id = '<root>'

-- per span, grouped
SELECT span_attributes.name, span_attributes.type, sum(metrics.prompt_tokens) AS pt, sum(metrics.estimated_cost) AS cost
FROM project_logs('<project_id>') WHERE root_span_id = '<root>' AND metrics.prompt_tokens IS NOT NULL
GROUP BY 1, 2

Observed on one trace (gpt-6.1-sol via openai.responses.create, 11 LLM calls):

Across 19 consecutive traces, the ratio of summary to leaf-llm sums was 2.000 for both tokens and cost in every trace.

Expected

A trace's rollup equals the sum of its leaf llm spans. Either the prompt span should not carry summed usage (or should carry it under a non-rolled-up name), or it should not be priced or summed as a billable call.

Workaround we use

We sum span_attributes.type = 'llm' AND span_attributes.purpose IS NULL spans and ignore the summary's token and cost metrics.