Repository navigation
fix(hermes): report native token usage - #4139
Merged
Merged
Conversation
Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
giuliolovisotto
added
bug
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
giuliolovisotto
marked this pull request as ready for review
October 6, 2026 22:20
giuliolovisotto
added
the
needs-review
ananthsub
added
the
r0.7.0
Contributor
|
seems to overlap with #3981 ? let's consolidate to one PR |
yaoyu-33
added
the
complexity:low
Contributor
|
Let's land #4139 first to unblock the release. It's a small, focused fix for Hermes reporting zero usage despite available native counters, and can be reviewed independently of the broader lifecycle accounting in #3981. I've already brought this native mapping into #3981. Once #4139 lands, I'll sync #3981 with main so its remaining diff covers the additional capture-based accounting and follow-up corner cases, including missing provider usage, auxiliary/delegated calls, reasoning details and cancellation handling. We can keep those follow-ups in #3981 without holding up this narrower fix. |
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/37662053501 |
ananthsub
pushed a commit
that referenced
this pull request
Oct 7, 2026
) beep boop [🤖]: Hi @giuliolovisotto 👋, we've cherry picked #4139 into `r0.7.0` for you! 🚀 Please review and approve this cherry pick by your convenience! Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com> Signed-off-by: NeMo Bot <nemo-bot@nvidia.com> Co-authored-by: Giulio Lovisotto <glovisotto@nvidia.com>
yaoyu-33
pushed a commit
to yaoyu-33/Gym
that referenced
this pull request
Oct 12, 2026
…o#4040) Pinned Hermes replays internal tool-call fields when it asks for a summary at the iteration limit. Gym rejects that request with HTTP 422, and the summary call bypasses Gym's observed model-call path. Delegated `delegate_task` children construct fresh Hermes agents, so they need the same adapter; otherwise their limit summaries fail and lack child invocation ownership. Keep the existing `cmunley1/hermes-agent@26bb847a` pin. Gym now installs its summary adapter before root and delegated child agents run, sanitizes only the outgoing summary history, and routes summaries and retries through the observed model-call method. The original trajectory stays intact and no turns are added. A documented `on_iteration_limit_reached(*, invocation_id: str) -> None` observer hook records `stop_reason=max_iterations` and `incomplete` once on the affected invocation before summary dispatch. A child's limit does not mark or terminate its parent. The sandbox observation bridge preserves this invocation-scoped outcome. No hook registry, budget change, or dependency repin is introduced. This addresses the follow-up to NVIDIA-NeMo#3554 and is tracked by [FEA-399](https://linear.app/nvidia/issue/FEA-399/hermes-observe-limit-summaries-and-invocation-stops). Native usage accounting is supplied by NVIDIA-NeMo#4139 on main. The adapter uses private methods of this pinned Hermes version, so a future dependency update must recheck these hooks. No configuration migration is needed. A valid terminal summary may change the final trajectory and benchmark result. Validation after rebasing onto main `4a9236e48c82f0dc23365a4c0088b1e54d00d3a8` (head `eb435d5da130538e2d87cd9ca0c384444aba9c54`): - Hermes component suite plus shared observability tests: **293 passed, 6 skipped**. The delegated-child regression uses Hermes's pinned child constructor in both host and sandbox observer paths and both installation orders. It verifies strict Gym schema acceptance, child-owned summary response IDs and limit outcome, and unchanged parent state. Other tests cover summary-error independence, one notification per limit, and sandbox observation import. - `pre-commit run --all-files` and `git diff --check` passed. All commits have DCO sign-off. Rebase compatibility changes are limited to the supervisor import and the sandbox observation test’s required `runtime_info` argument. - Fresh controlled conformance suite: `python scripts/run_harness_conformance.py --harness hermes --output <new-directory>` passed **7/7 scenarios** on the rebased head, including retry, omitted usage, terminal model error, and verifier failure scenarios. - Earlier controlled HTTP check: pinned Hermes → live Gym Model Server → scripted upstream returned HTTP 200 for a forced root summary, with an owned response ID and captured call. Outgoing tool calls had no internal fields. Script: `/tmp/hermes-summary-http-smoke-gym.py`. - Real-model root-summary smoke: HSG job `7676935` completed against earlier Gym commit `030307b48d4267a58b5c952cde1f0dd758b56a3c` on Super 3.5 direct P/D serving with `max_turns=1`, observability and model-call capture. Initial and limit-summary calls returned HTTP 200 and belong to the root invocation, with no trajectory gaps. The response is `incomplete` with `stop_reason=max_iterations` and no error. The verifier applied a patch and graded it unresolved; cleanup job `7676936` found zero sandboxes. Its summary response was reasoning-only, so Hermes appended no substantive assistant summary. Evidence: `/lustre/fsw/portfolios/nemotron/users/glovisotto/nemo-evaluator-rundirs/super-3dot5-pd/swe-bench-pro-hermes-summary-smoke/20261005_102034-a219dd2ebd76f36d/results` on HSG. This predates the child and invocation-hook changes, so it validates only the root summary path. - Real-model delegated-child smoke before this rebase: HSG job `7755765` completed against `5a6784a69c7cb8e13ee7643db14469cfddd87b1f` on Super 3.5 direct P/D serving with explicit delegation tools, parent cap 3, child cap 1, observability, and model-call capture. The model created `root.child-1` through `delegate_task`; all six calls returned HTTP 200, with two response IDs owned by the child and four by the root. The child is `incomplete` with `stop_reason=max_iterations`; the parent made two more model calls after the child returned, then independently reached its own limit and has the same stop reason. The rollout has no failure row or observation/trajectory gaps. The verifier applied a patch and graded it unresolved; cleanup job `7755766` found zero remaining sandboxes. Capture request bodies contain no internal `call_id`/`response_item_id` fields. Evidence: `/lustre/fsw/portfolios/nemotron/users/glovisotto/nemo-evaluator-rundirs/super-3dot5-pd/swe-bench-pro-hermes-child-summary-smoke/20261006_193101-3549a070ecb3cafb/results` on HSG. Health flags only the separate zero-aggregate-usage mismatch tracked by NVIDIA-NeMo#3981 / [FEA-397](https://linear.app/nvidia/issue/FEA-397/hermes-report-native-token-usage). The [rebased CI run](https://github.com/NVIDIA-NeMo/Gym/actions/runs/38126348032) records validation of the current head. The live-model smokes above are historical, with their source commits stated explicitly; this rebase changes no intended runtime behavior. CI: core coverage, all eight server-suite shards, `Nemo_CICD_Test`, lint, and DCO passed on the rebased head. CI conformance: Hermes passes **7/7** on this head. The overall P0 check remains red for OpenCode (2 failed scenarios), Pi (1), and Codex (7), matching the [current-main baseline](https://github.com/NVIDIA-NeMo/Gym/actions/runs/38123997566) exactly. Those failures are outside this Hermes summary fix. --------- Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hermes returned zero input and output tokens for every rollout, even when the pinned native harness supplied nonzero usage. Export its cache-inclusive
prompt_tokensandcompletion_tokensthrough the shared host/sandbox response converter, with cache and reasoning details. Missing native totals now remain unknown instead of becoming zero. This preserves the provider-failure and partial-result behavior established in #3720 and #3554.No linked issue: this fixes a token-accounting failure found during conformance validation.
Validation:
Seven regression cases failed against the original converter and pass with this change; they cover cache-inclusive totals, partial results and missing usage/details.
gym env test +entrypoint=responses_api_agents/hermes_agent +should_validate_data=true +skip_venv_if_present=true: 222 passed, 4 skipped.pre-commit run --all-files: passed.Final-head CI passed, including core tests, all eight server-test shards, lint and DCO.
Real ReasoningGym
manipulate_matrix, seed 42, one task/repeat with the same model and native verifier: reward 1.0, health healthy, 1,235 input / 376 output tokens. Original run was unhealthy because its response reported zero tokens. Capture and observability were enabled; the live agent source hash matches this commit.Seven fresh controlled scenarios through the native adapter: five healthy, one correctly unhealthy for intentionally omitted provider usage, one unobserved terminal-error sidecar. Before the fix: zero healthy, six unhealthy, one unobserved. P0 conformance remains incomplete for missing retry/terminal-error evidence.
Real native SWE-Pro NodeBB sandbox rollout: 91 model calls, healthy, no findings or unobserved rollout checks. Independent capture reconciliation matched all 1,874,133 input / 4,611 output / 1,830,400 cached input tokens. The native verifier completed with reward 0: the model supplied a plan and only created a backup file; the required
getOrCreate()implementation remained missing. This validates accounting, not task correctness.Sandbox run used the pinned native Hermes revision
26bb847a88493342ca1b194e0455b479073ae21dand task imagesha256:605b6da16030d20129e0b9f9376fd494d60ec5f68f5e05f608742133bd39ae98. A Linux-prepared runtime was mounted to meet the documented host/sandbox architecture requirement, and the existingchat_template_kwargs_enabled=falseoption was used for hosted GPT-4.1. Agent/resource/verifier code and task contents were unmodified. The all-P0 artifact assessment still lacks sandbox lifecycle evidence; healthy does not imply full conformance.This exposes the aggregates produced by pinned Hermes. It does not repair native omissions from auxiliary or delegated calls, nor infer missing usage from the health check's capture data. Both live runs exercised ordinary root calls; the SWE-Pro run completed before its 100-turn limit, so limit-summary and delegated-call accounting remain unvalidated.
Review coordination: a maintainer requested consolidation with #3981, which derives usage from shared lifecycle capture and leaves direct
/v1/responsesusage null. This narrower implementation exports native Hermes aggregates without requiring capture. The approaches overlap and should be consolidated deliberately; the live results above validate this PR's implementation, not #3981.