[gwarmstrong]

What does this PR do?

parse_stream_json added the result event's usage, which is already the run's cumulative total, to the usage of every assistant event, and stream-json repeats a message's usage on each of its content-block events. A Terminal-Bench-Science trial (Claude Code 2.1.207) reported 44.7M input tokens for a run whose result event says 15.6M.

The result event's usage is now the total. Without one (the CLI was killed or timed out), each assistant message counts once, with its latest usage; subagent messages still count, as before.

No issue: found in Terminal-Bench-Science runs; the numbers are above.

Validation

• claude_code_agent: 163 passed, including two new cases: a run with a result event, and a killed run without one whose content-block events repeat each message's usage. pre-commit clean.

Rollouts

N/A (usage accounting).

Compatibility and benchmark impact

Reported token usage for Claude Code runs drops to the real totals; rewards are unchanged.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[gwarmstrong]

/ok to test 85fe581

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5158-85fe58102c99.docs.buildwithfern.com/nemo/gym

[svcnvidia-nemo-ci]

SHIP WITH CARE

No implementation defect found in the result-total selection, interrupted-run deduplication, or downstream response construction. Both added tests provide concrete regression oracles.

RISK — Document the required real-smoke validation

The PR lists unit tests and marks rollouts N/A. However, the corrected counters are published as runtime response.usage by _response_from_output. The protected AGENTS.md requires representative real smoke rollouts for behavior-changing agent work; its metadata exemption covers catalog or manifest changes, not runtime accounting. The reported Terminal-Bench-Science discrepancy establishes the original problem but does not establish validation of the correction at this head.

Correction: Record a representative real-model smoke rollout using this head, inspecting the native stream, resulting Gym response, and verifier outcome. No implementation patch is indicated by this finding.

Preservation and acceptance: For a completed run, demonstrate that response input/output counts equal the native result counters and that total_tokens equals their sum, without adding assistant usage again. Confirm that the same stream retains its assistant text and tool-result sequence and that verifier behavior remains appropriate for the task. Retain the assertions (60, 9) and (90, 3) in the completed-run and interrupted-run regressions, respectively.

Review completed by static inspection; tests and rollouts were not executed here.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[lafrancef]

@gwarmstrong Thanks! To address the autobot you may consider temporarily adding claude code to the conformance checks: https://github.com/NVIDIA-NeMo/Gym/blob/main/scripts/harness_conformance/registry.py - this will exercise the harness under specific smoke test scenarios without starting a real run. You don't need to merge that specific change.

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40736195282

[gwarmstrong]

Smoke on this head (85fe581, in a build on main c155b77) with Claude Code 2.1.207 against a self-hosted DeepSeek V4.1 Flash served by vLLM: two Terminal-Bench-Science trials that ran to a 30 minute cap and were graded, plus a short local session that ends on its own so its stream has a result event.

The double count is gone: on the completed session's stream the old parser gives 110,447 input tokens, against a real 66,169. Two gaps remain. The result event splits input the Anthropic way (input_tokens 249, cache_creation_input_tokens 22,016, cache_read_input_tokens 43,904), so taking only its input_tokens reports 249. And vLLM's Messages stream puts output in message_delta, so the per-message usage in stream-json has output_tokens: 0, and a run killed at its cap reports 0 output (the two capped trials reported 5.7M and 7.3M input, 0 output). A fix is in progress: count both cache fields in input (and cache_read_input_tokens as cached_tokens), and stop reporting 0 output for runs killed before their result event. Rewards and grading are unaffected; only the reported token usage is wrong.

[gwarmstrong]

Follow-up to the smoke note above: the fix is in #5227. This PR's change stays: it removed a real double count, since stream-json repeats a message's usage on every content block.

What this PR got wrong, measured against a recording proxy between Claude Code 2.1.207 and a self-hosted DeepSeek V4.1 Flash:

• Cached input dropped. Input came from the result event's usage.input_tokens, which is only the uncached part. Cache writes and reads were dropped and cached_tokens stayed 0, so a five-call session reported 341 input tokens for 99,541.
• Subagent calls dropped. That usage covers the main conversation only. Subagent calls are in the result's modelUsage. In a two-subagent session it held 40,016 of 101,969 input tokens.
• No output on capped runs. A run cut off at the timeout has no result event, and on vLLM the per-message usage in stream-json has no output, so capped runs reported 0 output.

The tests here used hand-written events with no cache fields and with output in the per-message usage, so they encoded the same assumptions.

The follow-up does three things:

• It takes modelUsage for runs that end on their own.
• For capped runs, it counts the calls that finished, with their output, from Claude Code's session transcripts.
• It sets cached_tokens.

Its tests are built from real sessions, with expected values computed from what the server sent.