feat(google): preserve detailed usage metadata - #403
Conversation
427b09b to
12a0118
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12a0118e29
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| } | ||
|
|
||
| fn openai_chat_usage_view(usage: &Value) -> OpenAIChatUsageView { | ||
| serde_json::from_value::<OpenAIChatUsageView>(usage.clone()).unwrap_or_default() |
There was a problem hiding this comment.
When an otherwise valid Chat Completions usage object contains a malformed optional field such as "prompt_tokens_details": "invalid", deserializing the whole view fails and this fallback silently replaces every valid top-level counter with None; the previous field-by-field parser preserved prompt_tokens, completion_tokens, and total_tokens. Return a contextual error or model malformed optional details without discarding the rest of the usage.
AGENTS.md reference: AGENTS.md:L255-L259
Useful? React with 👍 / 👎.
12a0118 to
9754e5f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9754e5f196
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| prompt_tokens: Option<i64>, | ||
| completion_tokens: Option<i64>, | ||
| total_tokens: Option<i64>, |
There was a problem hiding this comment.
When a Chat Completions/Mistral usage payload has any malformed top-level counter (for example completion_tokens: "8" while prompt_tokens and total_tokens are valid), these plain Option<i64> fields make serde_json::from_value::<OpenAIChatUsageView> fail and the helper below returns Default, so all valid usage counters are silently dropped; the previous field-by-field parser would only ignore the bad field. Fresh evidence beyond the earlier optional-details issue is that the top-level counters still lack the tolerant deserialize_optional_i64 used for Anthropic, so please apply per-field tolerant parsing or return a contextual error instead.
AGENTS.md reference: AGENTS.md:L255-L259
Useful? React with 👍 / 👎.
9754e5f to
12ac79d
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 12ac79db91
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| // In the universal format, completion_tokens includes reasoning (matching OpenAI convention). | ||
| // Google separates candidatesTokenCount and thoughtsTokenCount, so we add them. | ||
| completion_tokens: Some(candidates + thoughts), | ||
| total_tokens: usage.total_token_count, |
There was a problem hiding this comment.
When a Google response omits totalTokenCount but still includes promptTokenCount, toolUsePromptTokenCount, candidatesTokenCount, or thoughtsTokenCount (all modeled as optional in UsageMetadata), this leaves UniversalUsage.total_tokens as None even though the aggregate is computable from the component counts. Other provider parsers added in this change fall back to prompt plus completion, so callers that serialize or inspect the universal response directly will lose the total only for Google payloads with an absent total; compute the same fallback here after deriving the inclusive prompt and completion counts.
Useful? React with 👍 / 👎.
12ac79d to
4d97054
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4d97054287
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| }), | ||
| ProviderFormat::Converse => { | ||
| let mut map = serde_json::Map::new(); | ||
| map.insert("inputTokens".into(), serde_json::json!(provider_prompt)); |
There was a problem hiding this comment.
When a universal usage object comes from OpenAI/Responses, prompt_tokens is already inclusive of cached/cache-write tokens; this line still writes that inclusive value as Converse inputTokens while the new code below also emits cacheReadInputTokens/cacheWriteInputTokens. The Converse importer treats those cache buckets as excluded from inputTokens and adds them back, so a Chat/Responses → Bedrock → OpenAI roundtrip double-counts cached prompt tokens; use exclusive_prompt_tokens() for Converse inputTokens before emitting cache fields.
AGENTS.md reference: AGENTS.md:L184-L189
Useful? React with 👍 / 👎.
4d97054 to
ff361db
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff361dbead
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| cache_creation: usage_token_breakdown( | ||
| input_details.cache_write_tokens, | ||
| None, | ||
| ), |
There was a problem hiding this comment.
When a Responses streaming event includes input_tokens_details.cache_write_tokens, this new universal input_details.cache_creation field is populated, but the Google streaming serializer has no cache-write usage field and drops it. The non-streaming expected-difference file accounts for Responses→Google, but streaming_expected_differences.json still only whitelists usage.prompt_cache_creation_tokens, so the cross-provider streaming guard will report an unexpected usage.input_details.cache_creation diff for those cases; add the matching narrow streaming expected difference or preserve the field through Google.
AGENTS.md reference: AGENTS.md:L142-L146
Useful? React with 👍 / 👎.
406219d to
5db4064
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5db4064c9b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| if inclusive_prompt.is_some() || self.completion_tokens.is_some() { | ||
| map.insert( | ||
| "totalTokenCount".into(), | ||
| serde_json::json!(prompt + completion), | ||
| ); | ||
| map.insert("totalTokenCount".into(), serde_json::json!(total)); | ||
| } |
There was a problem hiding this comment.
When callers use the public UniversalUsage::to_provider_value(ProviderFormat::Google) with a usage value restored from the new universal JSON shape that only has input_details/output_details, this Google arm still only writes the aggregate totals and later reads legacy fields such as self.prompt_cached_tokens/self.completion_reasoning_tokens, so cachedContentTokenCount, thoughtsTokenCount, and modality/detail buckets are dropped. The Google adapter uses the new UsageMetadata::from(self) path, but this public conversion helper should do the same or reuse the detail-aware locals already computed above.
AGENTS.md reference: AGENTS.md:L11-L13
Useful? React with 👍 / 👎.
5db4064 to
2c72543
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2c72543396
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| if let Some(cache_write) = cache_creation_tokens { | ||
| map.insert( | ||
| "cacheWriteInputTokens".into(), | ||
| serde_json::json!(cache_write), |
There was a problem hiding this comment.
Fresh evidence beyond the earlier Converse usage comment is that the current Bedrock streaming serializer still deserializes usage.to_provider_value(...) into BedrockTokenUsage at crates/lingua/src/providers/bedrock/adapter.rs:701, and that type only has input/output/total fields at crates/lingua/src/providers/bedrock/response.rs:229, so usage-only streaming chunks with prompt_cached_tokens or prompt_cache_creation_tokens discard the cacheReadInputTokens/cacheWriteInputTokens inserted here before the metadata event is serialized. Non-streaming Bedrock now preserves these buckets, but streaming Bedrock roundtrips still lose cache usage; extend the Bedrock streaming usage type/path to carry the new fields.
AGENTS.md reference: AGENTS.md:L11-L13
Useful? React with 👍 / 👎.
2c72543 to
a9665b9
Compare
There was a problem hiding this comment.
💡 Codex Review
Do not hard-code stop on metrics-only chunks
For Bedrock Anthropic streams that end with tool_use or max_tokens, the captured stream already carries that stop reason on the preceding message_delta while the following message_stop only adds amazon-bedrock-invocationMetrics (e.g. payloads/snapshots/toolCallRequest/bedrock-anthropic/response-streaming.json:96-109). This new metrics branch emits a second terminal choice with finish_reason: "stop", so OpenAI/Responses transforms can overwrite/report the final reason as stop and duplicate usage; make the metrics event a usage-only chunk or preserve the actual prior stop reason instead.
AGENTS.md reference: AGENTS.md:L11-L11
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
Google reports aggregate usage, per-modality counts, and a separate tool-use prompt bucket. Tool
prompt tokens are input tokens but are not included in `promptTokenCount`, even though they are
included in `totalTokenCount`:
prompt_tokens = promptTokenCount + toolUsePromptTokenCount
completion_tokens = candidatesTokenCount + thoughtsTokenCount
reasoning_tokens = thoughtsTokenCount
total_tokens = prompt_tokens + completion_tokens
A captured code-interpreter response demonstrates the accounting:
9 prompt + 96 tool prompt + 55 candidate + 32 thoughts = 192 total
Preserve the complete breakdown through provider-independent modality types and tool-prompt fields:
Google UsageMetadata
├── promptTokensDetails[]
├── cacheTokensDetails[]
├── candidatesTokensDetails[]
└── toolUsePromptTokenCount + details[]
│
▼
UniversalUsage
Subtract the retained tool-prompt subset when converting back to Google so both native fields
roundtrip without double-counting. Derive UniversalUsage serialization so all fields have one source
of truth, and document intentional losses for providers without equivalent usage fields.
Google UsageMetadata reference:
https://ai.google.dev/api/generate-content#UsageMetadata
a9665b9 to
3a9fcce
Compare
2 participants
Summary
Preserve detailed Google token usage without introducing provider-specific marker fields in the universal format.
This PR:
input_detailsandoutput_detailstoUniversalUsage.TokenBreakdown,InputTokenDetails, andOutputTokenDetailstypes.total_tokens, with an aggregate-derived fallback when absent.Google token accounting
Google reports tool-use prompts and thoughts separately from its normal prompt and candidate counts:
The captured code-interpreter response demonstrates this accounting:
When converting back to Google, the retained tool-prompt subset is subtracted from canonical prompt tokens and reasoning is subtracted from canonical completion tokens. This restores the native buckets without double-counting.
Universal representation
Cache, tool-prompt, and reasoning counts remain subsets of the aggregate input/output totals. Modality counts remain breakdowns rather than additional totals.
Validation
cargo test -p lingua— 1,216 passedmake test-payloads— 1,730 passed, 65 skippedmaingateway compiled against this Lingua branch without Braintrust changescodeInterpreterToolParamcoverage: 80/91; remaining failures are existing Bedrock limitationsGoogle
UsageMetadatareference: https://ai.google.dev/api/generate-content#UsageMetadata