Repository navigation
fix(ai): report tokens used by an interrupted stream instead of zero - #4664
Conversation
The OpenAI wrapper accumulates token counts as chunks arrive. When a stream failed or was cancelled partway it called the accumulator, took the completion id and system fingerprint off the result, and then reported hardcoded zeros for usage and latency. A call that consumed a large prompt was recorded as costing nothing. The error path now passes on the counts the accumulator collected, plus the elapsed latency. Token counts are also left absent when the provider never reported any, rather than defaulting to zero, so a stream that carried no usage is no longer indistinguishable from one that consumed nothing. This matches how output tokens were already handled. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
|
Size Change: +7.03 kB (+0.03%) Total Size: 20.8 MB 📦 View Changed
ℹ️ View Unchanged
|
The Anthropic, Gemini, Azure and Vercel wrappers carried the same defect as the OpenAI chat path: an error or cancellation reported hardcoded zero token counts, and often a latency of zero, for a call that had already consumed its prompt. Where a stream accumulates usage, the error path now reports what it collected and the elapsed latency. Anthropic sends input tokens on message_start and Gemini sends usageMetadata on chunks, so a stream cut short usually carries real counts. Where no usage exists, such as a request that failed before any response, the counts are now absent rather than zero. Absent says the call may have consumed its prompt without reporting it; zero asserts it consumed nothing. Also covers the OpenAI Responses and transcription streams and the Responses accumulator, which had the same shape as the chat path already fixed. Four tests asserted the previous zeros on error paths. They now assert the counts are absent. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Versioning CI requires the PR title's conventional-commit type to match the changeset bump level, and rejected a fix title paired with a minor bump. Patch is the accurate pairing. The package API is unchanged: no exported type or signature moves. What changes is the shape of the event data the wrappers emit, which the changeset body calls out for anyone reading $ai_input_tokens downstream. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
A costOverride multiplied its prices by token counts defaulted to zero, so a call whose provider never reported usage sent $ai_total_cost_usd: 0. Ingestion rightly trusts a client-sent total, so that zero went straight to the UI as a free call — recreating, for costOverride users, the exact symptom this branch removes. Each component is now priced only when its token count exists, and the total sums the priced components. A call with no reported usage sends no cost at all, and a partially known call sends the known side. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Five error captures still sent a hardcoded latency of 0, which is the same class of wrong as a zero token count. Every one had startTime in scope, so they now report the elapsed time. The error builders in telemetry.ts require latency instead of defaulting it to 0, which made the compiler find every caller, including the Azure chat stream error path that was also dropping the accumulated usage the OpenAI wrapper already passes. The headline behavior gains its missing tests: both Anthropic stream-safety tests now yield a message_start carrying input usage before the interruption and assert the captured event reports those tokens with the output count absent, and a new pure test pins that the error builders carry the caller's usage and latency into the options. The changeset and the cost comment now state what $ai_total_cost_usd means when only one side was reported: the cost of the known side alone, a lower bound rather than an assertion. The rationale for optional counts, repeated across some twenty comment blocks, now lives once on TokenUsage. Call sites that restated it, or restated what the adjacent code already shows, lost their comments. Generated-By: PostHog Desktop Task-Id: 5cb23cd6-224c-4d86-bc24-97730d4c2f98
|
@carlos-marchal-ph leaving this to you since it looks like more of a bigger change that maybe would need to be fixed in other SDKs as well |
|
@marandaneto yeah sure thing! @bernatixer joined us last week, so he's taking larger and larger tasks step by step, but I agree that it might be interesting to do a bigger picture assessment of whether this impacts other SDKs. |
oh til, i thought it was a community PR :D |
A consumer that breaks out of a Gemini stream resumes the pending yield as a return, so neither the loop tail nor the catch ran and a cancelled stream emitted no event at all. Only a finally observes that path, so the success capture moved into one, behind an errored flag so the error path still captures exactly once. Cancellation now reports the accumulated usage, matching what the changeset already claimed. The Anthropic error path dropped the raw usage collected from message_start, because usage.rawUsage was only assigned on clean completion. The catch now pins whatever raw usage arrived, and stays silent when the stream died before message_start. The changeset is trimmed to two sentences for the changelog. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
$ai_total_cost_usd reads downstream as the whole cost, but a one-sided cost override sent the known side's cost as the total: a lower bound presented as a total. The total is now written only when both sides were priced, mirroring how token counts are omitted when unknown. A partial call sends its components alone. Ingestion applies the same rule on its side, so the semantic holds whether the total is computed here or there. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Reverts the both-sides-only total. PostHog's AI credit usage report bills only positive values of $ai_total_cost_usd, and trace-level views sum event totals, so omitting the total on a one-sided call drops its known cost from billing and from trace headers. A lower bound is the least wrong value for a property every consumer sums; the semantic stays documented at the write site. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Hey thanks for the review! I checked and we have the same behavior on the Python SDK utils.py#L784-L785 |
Every integration wrote $ai_input_tokens: 0 and $ai_output_tokens: 0 when the provider never reported usage, so an interrupted stream priced as a free call downstream. Token counts are now written only when they trace back to a provider report: the shared capture paths, both embedding events, the LangChain callback and the Agents and Claude Agent SDK processors omit unreported counts, and the stream accumulators and response extractors no longer seed at zero. A zero the provider reported is still sent as 0, so zero keeps meaning a real report of nothing. LangChain cache and reasoning counts follow the same rule instead of coalescing to 0, and the Agents processor's $ai_total_tokens is the sum of the reported sides, omitted when neither side reported. Mirrors the posthog-js change (PostHog/posthog-js#4664). Ingestion distinguishes absent from zero since PostHog/posthog#90211 and older ingestion coalesces absent to 0 at read time, so this ships independently. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
* fix(ai): omit token counts the provider never reported Every integration wrote $ai_input_tokens: 0 and $ai_output_tokens: 0 when the provider never reported usage, so an interrupted stream priced as a free call downstream. Token counts are now written only when they trace back to a provider report: the shared capture paths, both embedding events, the LangChain callback and the Agents and Claude Agent SDK processors omit unreported counts, and the stream accumulators and response extractors no longer seed at zero. A zero the provider reported is still sent as 0, so zero keeps meaning a real report of nothing. LangChain cache and reasoning counts follow the same rule instead of coalescing to 0, and the Agents processor's $ai_total_tokens is the sum of the reported sides, omitted when neither side reported. Mirrors the posthog-js change (PostHog/posthog-js#4664). Ingestion distinguishes absent from zero since PostHog/posthog#90211 and older ingestion coalesces absent to 0 at read time, so this ships independently. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf * chore(ai): update public API snapshot for Optional embedding count extract_gemini_embedding_token_count now returns Optional[int], returning None when no embedding carried a token count. Regenerated with make public_api_snapshot. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
capture_streaming_event defaults auxiliary token fields to zero: $ai_cache_read_input_tokens and $ai_reasoning_tokens for OpenAI, both cache fields for Anthropic. A stream interrupted before the provider reported any usage therefore sent fabricated zeros, and cost processing reads a reported zero as a known count, pricing the generation at a known $0.00 instead of leaving it unknown. #906 fixed the main token counts on this path but missed these fields. The zero-defaults now apply only when the provider reported usage at all (input or output tokens present). Completed and partially reported streams keep the historical zeros, so nothing changes for them. This matches the JS SDK, which omits all usage fields on interrupted streams since PostHog/posthog-js#4664. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
capture_streaming_event defaults auxiliary token fields to zero: $ai_cache_read_input_tokens and $ai_reasoning_tokens for OpenAI, both cache fields for Anthropic. A stream interrupted before the provider reported any usage therefore sent fabricated zeros, and cost processing reads a reported zero as a known count, pricing the generation at a known $0.00 instead of leaving it unknown. #906 fixed the main token counts on this path but missed these fields. The zero-defaults now apply only when the provider reported usage at all (input or output tokens present). Completed and partially reported streams keep the historical zeros, so nothing changes for them. This matches the JS SDK, which omits all usage fields on interrupted streams since PostHog/posthog-js#4664. Generated-By: PostHog Desktop Task-Id: ec0cc27f-fc40-4994-b153-2bfe74de1edf
Problem
A user cancels a streaming response, and the generation is recorded as costing nothing. The call consumed its whole prompt, so the cost is real; only the report is wrong.
Every provider wrapper accumulates token counts as chunks arrive. When a stream failed or was cancelled partway, the error path reported hardcoded zeros for usage, and usually a latency of zero, discarding counts it had already collected. In the OpenAI chat path the counts were read one line before being thrown away.
A second path produced the same result:
captureAiGenerationsent$ai_input_tokens: usage.inputTokens ?? 0, so a generation whose usage was never reported still claimed zero input tokens.Changes
message_startand Gemini sendsusageMetadataon chunks, so a stream cut short usually carries real counts.$ai_output_tokenswas already omitted when absent. Input tokens now match it.costOverrideprices only the token counts the provider reported. A call with no reported usage sends no cost instead of$0.00, and a partially known call sends the known side as a documented lower bound, because PostHog's billing and trace views sum this property. Omitting the total was tried and reverted for that reason.finallyobserves, so the success capture moved into one.message_starton the error event, alongside the token counts.Warning
$ai_input_tokenswas always present on a generation event before this change. Anything reading it has to handle it being absent.Release info Sub-libraries affected
Libraries affected
How did you test this code?
@posthog/aisuite. It passes, with two pre-existing failures inopenai-agents-resolution.test.tsthat reproduce identically on an unmodified tree in this sandbox.posthog-nodeand@posthog/corelocally was necessary before eight suites, including the Anthropic and Gemini ones, could run at all. They were silently unloadable before that, and running them found four assertions this change breaks.openai-stream-accumulators.test.ts. The regression it catches: someone re-seeds an accumulator's token counts at zero, and every cancelled stream silently reports as free again.tscoutput before and after: identical once line numbers are normalized.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Written by Claude Code, directed by a PostHog engineer investigating why a customer's traces showed no cost.
The customer turned out not to use this package, so this change does not fix their case. It fixes the same class of defect for the users who do. The matching change in the main repo is PostHog/posthog#90211, which stops the ingestion pipeline pricing an absent token count as zero. The two are complementary: without this one, the zeros this package sends would keep that fix from ever taking effect for its users.
The first revision covered only the OpenAI chat path and stated that the other providers were left alone. They are all covered now, at the reviewer's request.
Public artifact: no customer data, project identifier, or operational figure appears in this PR, its commits, or its tests. The reproduction used a mock provider and an invented prompt.