Repository navigation
Conversation
GDPVal is wired to stirrup_agent, which drives the model itself, so a GDPVal score says nothing about the harness under test. This agent provisions the GDPVal container, stages the task's reference files, runs another Gym harness inside it and collects the files that harness produced, which makes the benchmark usable for harness comparison. The harness has no tool for declaring its deliverables, so the prompt names an output directory and only that directory is collected; if it is empty, files newer than a start marker are swept instead. The sandboxed harness reaches the model server over the host network, so its calls carry the rollout prefix and the rollout id is handed to the harness as path_params. Without them a harness takes an uninstrumented path that reports zero tokens and emits no observations, and the model server has no rollout to file the calls under. The harness run is timed and reported as agent_run_time so it can be measured. Signed-off-by: James Nolan <jnolan@nvidia.com>
vision_analyze only registers when a vision route resolves. The probe asks whether a vision client could be built, and it runs before the live endpoint is recorded, so a self-served checkpoint absent from the models.dev catalogue fails twice over: no declared capability, and no endpoint to construct a client from. It then falls through to third-party vision providers we hold no credentials for, and the model is told the tool does not exist. Separately, hermes picks its non-stream stale timeout from an allowlist keyed on the model name, which never matches here because the name it receives is model_server.name, so long reasoning turns are abandoned at the 90s chat default. Both are config fields defaulting to the previous behaviour: a text-only model is not declared able to read images, and a caller that never asked keeps the harness's own timeout. Signed-off-by: James Nolan <jnolan@nvidia.com>
A harness shows the model a picture by returning it from a tool, but tool content was
annotated as text parts only, so the request schema rejected the whole conversation
the first time one came back:
loc body.messages.N.NeMoGymChatCompletionToolMessageParam.content
.list[NeMoGymChatCompletionContentPartTextParam].1.type
msg Input should be 'text'
got image_url
Tool content now takes the same parts a user message may carry, like the name field
above it: the OpenAI type is narrower than the servers behind Gym.
Signed-off-by: James Nolan <jnolan@nvidia.com>
The verify response is built from the modelled request fields, so an unmodelled one is dropped on the way back and never reaches the rollout: a full run recorded agent_run_time on 0 of 1100 rollouts even though the agent measured it. Set it on the returned record instead of relying on the resources server to echo it. Signed-off-by: James Nolan <jnolan@nvidia.com>
At 5400s the sandbox killed 202 of 1100 rollouts on a full run. Each had already made 44 to 82 model calls, so the work was done and then discarded: the harness never wrote its response, the agent fell back to an empty one, and the judge scored nothing. Those rollouts averaged 0.30 against 0.84 for the ones that finished. stirrup_agent, which owns this benchmark, allows 3h30m and describes it as a guard against a task that would otherwise consume a whole Slurm walltime, not as a working budget. Use the same figure. Signed-off-by: James Nolan <jnolan@nvidia.com>
_collect truncates at 100 files. One task ran a package install inside the output directory, so the harvest competed 98 vendored OpenZeppelin sources against the model's own contract, README and circuit, and dropped the latter. Exclude the directory names a build leaves behind, so a dependency tree cannot enter the candidate set, and tell the model to build elsewhere. Across 1100 rollouts the cap binds once and only in that task, and the exclusion alone resolves it. Signed-off-by: James Nolan <jnolan@nvidia.com>
Hermes caps web_search at 50 calls per turn and resets the counter each turn, which never starves a normal multi-turn session. A GDPVal task runs as a single Hermes turn, so the cap acts as a whole-task ceiling instead, and tripping it halts the loop rather than warning. One task lost its run to it after 50 searches that had all succeeded with distinct content, with the document it was researching never written. Expose the cap on the Hermes agent config and raise it here to match max_turns. Not disabled: that would remove the runaway-loop protection entirely, where matching max_turns keeps it at about one search per iteration while removing the ceiling our one-turn design created. Signed-off-by: James Nolan <jnolan@nvidia.com>
nemo-automation-bot
Bot
added
the
community-request
yaoyu-33
added
area:agent
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
github-actions
Bot
added
the
sla:triage-overdue
svcnvidia-nemo-ci
added
the
waiting-on-maintainers
Contributor
nemo-automation-bot
Bot
added
the
community-request
yaoyu-33
removed
community-request
svcnvidia-nemo-ci
added
waiting-on-customer
yaoyu-33
added
the
complexity:medium
This was referenced Oct 22, 2026
svcnvidia-nemo-ci
added
waiting-on-maintainers
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Gym has a GDPVal implementation but it's wired to the stirrup harness, as per the GDPVal-AA-v2's methodology. Our team would like to run GDPVal using the Hermes harness. To do so, this PR adds a new GDPVal implementation where the harness can be swapped.
This new implementation differs from the original GDPVal as follows:
This PR doesn't aim to replace the existing GDPVal implementation. The generic implementation has a cost: the harness and its dependencies have to be provisioned into the container before the first task can run, which the host-side stirrup path avoids. It also diverges from AA-v2's methodology, which names stirrup as the harness, so a score from this path is not the published quantity.
Two changes fall outside the new directory.
hermes_agentgains two optional config fields,supports_visionandapi_call_stale_timeout, both defaulting to off. AndNeMoGymChatCompletionToolMessageParam.contentis widened to accept the same content parts a user message accepts, because a harness shows the model an image by returning it from a tool, and the text-only type rejects the whole conversation the first time that happens.The PR looks large, but a lot of it is copied from the existing cvdp implementation. The duplication is not ideal, but it looks to be a common pattern when adding a new benchmark/agent in gym:
setup_scripts/_portable_python.shalready exists in four agents with three different contents. TBD if there's a better approach or a plan to unify/reuse utilities.Checklist
pre-commit run --all-files) (so CI lint/format/copyright pass).git commit -s) (so the DCO check passes).