Repository navigation
feat(agent): run OpenClaw inside task sandboxes - #3637
Conversation
506e097 to
3209554
Compare
00db9de to
b5a692e
Compare
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Use shared lifecycle hooks, join and replay identical activations, isolate waiter cancellation, and propagate runtime errors before verification. Compose prepared rows through independent definitions; retain the upstream SWE-Pro scoring behavior. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
d634bfb to
04121ea
Compare
Use the same supervisor transport as Pi and Codex. Keep cleanup receipts, worker identity, and harness output separate while preserving borrowed/owned teardown and retry behavior. Validation: 314 passed, 38 skipped; full pre-commit passed. Linux-specific tests and fresh model rollouts remain pending. Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
ananthsub
left a comment
There was a problem hiding this comment.
Thanks for this. The sandbox-session path closely follows the Pi (#3626) and Codex (#3640) shape:
sandbox_providerdefaults toNone;sandbox_install_timeout_secondsand a 60-second close timeout;- a shared first-use local install off the event loop, with retry;
BaseExceptionhandling that keeps partial transcripts;- an installer that verifies checksums and accepts both the Debian and RHEL certificate paths;
stderr.logcollected as an artifact.
Whether an agent session exists picks the path, with no extra selector, which keeps the config simple.
Main requests:
- Drop "native" terminology. It appears in the PR title and description, the README, user-facing error messages, the installer, and the tests. The repo moved away from it in #4070. "Sandbox sessions" matches the other harnesses.
- Add a compatibility note to the PR description for the shipped
openclaw_agent.yamlchanges: thenvinfprovider and model are removed, several local keys are removed, andresources_serveris nownull. - Consider an allow-list for request validation, as Pi has, so that new Responses API fields are rejected by default.
Smaller items: the install timeout is 900 while Pi and Codex use 600; the local install moved from startup to first use; and the session helpers are duplicated across harnesses, which can be a follow-up.
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 01218cd |
ananthsub
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. All seven threads are resolved:
- "native" is gone from the title, README, error messages, installer and tests.
- The compatibility note and the first-use install note are in the README and the PR description.
- The install timeout is now 600.
- The request allow-list matches Pi's code exactly, with a test that rejects a newly added schema field.
- The shared helpers are tracked in #4306.
The OpenClaw tests pass locally on Linux.
New comments are all nits (two inline). The allow-list now rejects request metadata, including the seeded-repeat seed, and the chat_template_kwargs hint was lost. Also, the PR description says the request-field contract matches "Pi/Codex", but Codex still uses a deny-list; only Pi has the allow-list.
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
|
/ok to test 0ba7a55 |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/38022831491 |
…Mo#4476) Sandbox (native) OpenClaw sessions rejected the configured `max_output_tokens`, so the generated model entry had no `maxTokens` and OpenClaw used its own per-call output default. On a long-form agentic benchmark, a quarter of the judged deliverables ended truncated at that default while a comparable harness ran at 32768. Changes: - Accept `max_output_tokens` in sandbox mode and write it as `maxTokens` of the generated model entry, as the local CLI path already does. - Validate it (positive, not above `context_window`) at activation. - README: describe the behaviour; keep the note that metadata does not prove an effective inference limit and that an episode-wide budget is not implemented. Default stays unset, so existing configs behave exactly as before. Measured on a full 220-task run with the recipe setting 32768: truncated deliverables fell from 55 to 3 and the comparison ELO rose by about 95 points on the same tasks and judges. Mirrors yaoyu-33#13 onto main now that NVIDIA-NeMo#3637 is merged. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Adam Rajfer <arajfer@nvidia.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
What / expected
Run OpenClaw in borrowed or agent-owned task sandboxes through EnvironmentServer sessions, using the shared execute/close lifecycle. No agent session uses the existing local CLI. Invalid/closed sessions never fall back to the host; cleanup must finish before verification.
No separate feature issue: this completes the session integration requested here. Shared-helper/fixture cleanup is tracked in #4306.
Compatibility / concerns
The shipped YAML replaces the fixed
nvinfprovider/model with Gym'spolicy_modeland${policy_model_name}. Removed example keys (workspace_root,openclaw_agent_id,thinking,system_prompt,extra_args,env,concurrency) use class defaults unless overridden.resources_serverchanges from required to null: local/runstill needs an explicit binding; direct/v1/responsesdoes not.Local installation now happens on first use, off the event loop, with shared attempts and retry after failure—not at startup. Sandbox install/close budgets are 600/60s. Unsupported session controls and required external HTTP/MCP tools fail explicitly. Eval-only; one worker. Existing local usage accounting and outer-cancellation limitations are separate follow-ups. No benchmark recipes or accuracy baselines are migrated.
Session request
metadatais rejected, includingextra_body.seedfromnum_repeats_add_seed; disable that option for this path. Configurechat_template_kwargson the Gym model server; the rejection now explains this.Validation
Current head:
0ba7a55c7dab00208ff4ba4f0d3e086feef60e28; merged mainf2695eb4c.Pytest selection:
responses_api_agents/openclaw_agent/tests,responses_api_models/vllm_model/tests, and coretest_sandbox_session,test_sandbox_supervisor_client,test_agent_sessions,test_agent_session_retries,test_add_legacy_agent_environment_servers,test_global_config. Run with--import-mode=importlib -q -o addopts='' -o log_cli=false; Linux exclusions are named below.test_publish_rebinds_clients_and_clears_sessions,test_unpublished_endpoint_grace_lifecycle); those unchanged files match main. Canonical isolatedgym env testand core-wide coverage were not rerun.nvidia/nemotron-3-super-120b-a12b, borrowed Docker sandbox, stock math verifier: reward 1, 3 model calls, 13,214 tokens, paired tools, tool-written answer and retained observations. Normal model-server retries recovered two HTTP 429s within the same episode. Agent close confirms cleanup before verification; Resources then removes the container.Earlier revision-pinned evidence
At
c0136aee8, owned and borrowed model smokes scored 1 each. HSG SWE-Pro completed official grading with reward 0, 161/178 tests passed; the model hit its output limit without a patch. Atfa5ae88e1, direct/local and legacy/runrouting smokes scored 1; one direct response returned zero usage, while/runusage was positive. That local accounting gap is not fixed here. Earlier runs are not presented as validation of this head.