Repository navigation
fix(conformance): prefer agent sessions - #4489
Merged
Merged
Conversation
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
lafrancef
added
feature
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
lafrancef
changed the title
Contributor
Author
|
/ok to test 0b68751 |
Signed-off-by: Frederic Lafrance <flafrance@nvidia.com>
Contributor
Author
|
/ok to test 0bab21d |
yaoyu-33
added
complexity:medium
marta-sd
approved these changes
Oct 13, 2026
pachmu
pushed a commit
to pachmu/Gym
that referenced
this pull request
Oct 13, 2026
The harness conformance checks (NVIDIA-NeMo#4489) exposed a gap in observability: when an agent turn failed, the failure record carried none of the agent's observations, so a model-error episode left no agent evidence to check. The agent session was closed only in the final unwind, and its observations were discarded. This PR closes the agent session when the agent stage fails and carries its observations on `SingleAgentTurnFailure` (verification failures too). They reach the collected failure record as `ng_agent_observations`, which then gets the capture join and trajectory like a successful row. It also adds the `NG_AGENT_OBSERVATIONS_KEY` constant for that key. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Marta Stepniewska-Dziubinska <martas@nvidia.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
chtruong814
approved these changes
Oct 13, 2026
zyzhou5
added
ready-to-merge
zyzhou5
enabled auto-merge
October 13, 2026 19:11
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38452985512 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 13, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38531566635 |
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 14, 2026
Contributor
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38542378021 |
mk-0
added a commit
that referenced
this pull request
Oct 14, 2026
Sandbox sessions: - A run that times out before any model call completed (for example an unreachable model server) now fails instead of being graded as a zero; the error carries the tail of Kilo's log, which names the cause. - A retried close no longer cancels an activation that is already cancelling. - The runner records its hostname and PID; responses report them with the Kilo version, and the sandbox record carries the sandbox id. Local runs: - Output is file-backed and the process group is killed on timeout and on request cancellation, so partial events survive and Kilo never outlives its request. - CLI errors, nonzero exits and timeouts before any model call return a failed response with the reason; /run reads Kilo's session database and returns agent observations. Shared: - Use opencode_agent's session parser (maps bash outcomes from exit codes) instead of the deprecated opencode_sandboxed_agent. - Disable Kilo's auxiliary session-title call by default for Gym-managed providers. - Register Kilo in the harness conformance suite; rename setup_kilo to setup_kilocode to follow the setup_<harness> convention. - Add a read-only AsyncSandbox.sandbox_id property. Conformance (session mode, Linux): all checks pass in 6 of 7 scenarios; model_error shows the session-mode gap shared by all harnesses (#4489). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com>
7 tasks
7 tasks done
4 of 7 tasks
This was referenced Oct 19, 2026
giuliolovisotto
added a commit
that referenced
this pull request
Oct 20, 2026
Preserve agent-owned sandbox identity and grade initial HTTP 400 rejections through agent sessions after the conformance runner switch. Keep infrastructure and partial-generation failures exceptional. Refs #4489 Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
giuliolovisotto
added a commit
that referenced
this pull request
Oct 20, 2026
Restore native call identity, ownership and bounded 429 retries on the agent-session path selected by conformance since #4489. Grade an initial HTTP 400 rejection only after clean execution with no generated output. Signed-off-by: Giulio Lovisotto <glovisotto@nvidia.com>
ekellbuch
pushed a commit
to ekellbuch/Gym
that referenced
this pull request
Oct 20, 2026
* feat(kilocode): run Kilo in task sandboxes Add native sandbox sessions to the KiloCode agent so it composes with any Resources server through single_agent_turn_legacy, as Pi and Codex do. - Session setup installs a checksum-verified Node 22 and the pinned @kilocode/cli into a private per-session directory (glibc and musl); task images need only Linux and Python 3.8+. - The CLI runs under the shared SandboxSession supervisor in the task workdir with isolated HOME/config/database and no repo-level config. - Close returns observations from a snapshot of Kilo's SQLite database (root and child sessions, tool timing, usage), annotated with the run outcome; CLI/API errors fail the episode instead of scoring zero. - Kilo's stream watchdog (chunkTimeout) follows the harness timeout, since the Gym model server returns completions only once finished. - Host installation becomes lazy, on the first local invocation. - kilocode_sandboxed.yaml, SWE-bench Pro allowed_agents, and the legacy environment-server migration entry. Validated on SWE-bench Pro with the stock Resources server: 5-task smoke across Ubuntu/Node/Go/Python/Alpine images, and 731x3 at 51.30% mean pass@1 (Nemotron 3.5 Super). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com> * docs(kilocode): trim sandbox README to the shared harness layout Keep the sections other native harnesses document; move run results and limitations to the PR description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com> * fix(kilocode): align sandbox sessions with the reference harnesses Address review and match Pi, Codex, OpenCode and OpenClaw. - Classify a run from the supervisor cleanup receipt. A runner exit, supervision error or missing exit status without a timeout fails the episode instead of returning an empty incomplete response. Supervised timeouts and length stops stay gradable. - Run a POSIX sh bootstrap before the Bash installer, and check in the sandbox that the task workdir and session storage do not overlap after resolving symlinks. - Return 422 for unsupported setup, require direct SandboxAccess, and return 409 for activation of a closing session or /run with a session. - Snapshot the Kilo database only after confirmed cleanup, and report a gap when the receipt has no exit code. - Add bootstrap, lifecycle-contract and exec-transport tests from the reference harness templates. Validated on 250 SWE-bench Pro tasks with the stock Resources server: 250/250 graded, no agent or infrastructure failures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com> * fix(kilocode): surface failed runs and pass harness conformance Sandbox sessions: - A run that times out before any model call completed (for example an unreachable model server) now fails instead of being graded as a zero; the error carries the tail of Kilo's log, which names the cause. - A retried close no longer cancels an activation that is already cancelling. - The runner records its hostname and PID; responses report them with the Kilo version, and the sandbox record carries the sandbox id. Local runs: - Output is file-backed and the process group is killed on timeout and on request cancellation, so partial events survive and Kilo never outlives its request. - CLI errors, nonzero exits and timeouts before any model call return a failed response with the reason; /run reads Kilo's session database and returns agent observations. Shared: - Use opencode_agent's session parser (maps bash outcomes from exit codes) instead of the deprecated opencode_sandboxed_agent. - Disable Kilo's auxiliary session-title call by default for Gym-managed providers. - Register Kilo in the harness conformance suite; rename setup_kilo to setup_kilocode to follow the setup_<harness> convention. - Add a read-only AsyncSandbox.sandbox_id property. Conformance (session mode, Linux): all checks pass in 6 of 7 scenarios; model_error shows the session-mode gap shared by all harnesses (NVIDIA-NeMo#4489). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com> * fix(kilocode): keep the task's HOME for tool shells and retain setup state Kilo's tool shells inherited the HOME and XDG directories that keep Kilo's own state private, so files that tools wrote under the home directory landed in the session directory and were deleted by agent close before verification. A plugin now restores the task's HOME and XDG directories for every shell command through Kilo's shell.env hook, as OpenCode does. Sandbox runs no longer pass --pure, which would skip that plugin; codebase indexing stays off because kilo.json names no indexing plugin. Kilo only logs a plugin that fails to load, so a run without the plugin's receipt fails instead of being verified. Also from review: - A cancelled local caller no longer releases another caller to start a second installer; one shielded installation task is shared, as in Pi. - An owned sandbox is retained in the session before start(), so close can stop a sandbox that start() allocated but failed to finish setting up. - A child invocation that stopped on the output limit is reported as incomplete, not completed (require_terminal_finish=True). Validation: Kilo tests pass locally, including the plugin loaded by the pinned Kilo 7.4.15 binary; the Linux unit suite on AGA shows no new failures; a 5-task SWE-bench Pro smoke returned and graded 5/5 (4 solved), with the plugin confirmed in every run and no tool output referring to Kilo's private home. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Marat Kitov <mkitov@nvidia.com> * fix(kilocode): read kilo_version in conformance install and fix musl docs Conformance CI install looked up kilocode_version, but Kilo's configs only define kilo_version, so setup raised KeyError for the newly registered harness. Map kilocode to kilo_version. The README claimed musl support on aarch64, but install_kilo_runtime.sh rejects it (the pinned Node musl build is x86_64 only). Document that. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> * test(kilocode): align exec transport tests with main's termination semantics Main now leaves detached tools from a natural CLI exit for the sandbox owner to tear down, and treats a negative cleanup receipt as final: close releases the borrowed session instead of staying retryable. Update the KiloCode exec transport tests to match, as main did for Codex and OpenCode. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> --------- Signed-off-by: Marat Kitov <mkitov@nvidia.com> Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: yaoyu-33 <yaoyu.094@gmail.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Prefer the agent session interface in the harness conformance runner, with legacy agent
/runfallback when session setup/close are unimplemented. Follow-up to #3766.Detect implementations through the shared agent contract, including inherited handlers, and reuse
single_agent_turn_legacyEnvironmentServer orchestration. Migrating an adapter to that contract automatically switches its probes. Agent session setup, activation, and cleanup failures remain visible and never trigger legacy fallback. CI preserves the pinned uv executable on PATH for Hermes session setup in later steps.Validation
Reproduction commands for the Linux probes:
Rollouts
All four real harnesses produced agent session rollouts with verification and Resources cleanup. Pi and Hermes tool-success/failure probes passed P0. The measurements retain Codex trajectory/ownership gaps, Pi retry ownership gaps, OpenCode local-sandbox identity gaps, and terminal model-error failure sidecars.
Compatibility and benchmark impact
Probes using agent sessions require Linux and install pinned harness runtimes during session setup. Legacy probes retain their preinstalled runtime requirement. Runtime metadata records the selected interface and resolved version pins. Probe expectations and benchmark scoring remain unchanged.