[JeffPengCoder]

Summary

This PR adds first-class OSWorld support to NeMo Gym for both standalone evaluation and exact multi-turn RL training. It keeps one OSWorld task/evaluator path while allowing training callers to request the evidence needed to reproduce the model calls that actually drove the desktop.

Gym records the semantic trajectory and the real per-turn prompt, generation, token/logprob, media, identity, termination, and evaluator facts. A trainer such as NeMo-RL remains responsible for validating that evidence, converting it into trainable rows, choosing loss masks, computing advantages, and running backward. Gym never invents an append-only token history when a later multimodal prompt was independently rendered or compacted.

The runtime supports Gym Sandbox with local or remote Docker and a pre-provisioned OpenSandbox KVM Pool. The pinned OSWorld dependency retains canonical task configs and evaluator semantics; Gym owns orchestration and does not patch task setup at runtime.

Scope split and follow-ups

The branch stack is:

upstream main
  -> #2554 OSWorld runtime and exact-trace contract
     |-- #3139 sink-window history policy and failure taxonomy
     `-- #3140 AgentENV desktop through an E2B-compatible gateway

• #3139 Add the sink-window history policy
• #3140 Run OSWorld on AgentENV through E2B

Both follow-ups are based directly on this PR and are independent siblings. Because this head lives in a personal fork, they temporarily target upstream main; their diffs shrink to their own increments after #2554 merges. Transport-contract hardening is also kept out of this PR until it can be reviewed as a separate layer.

Design

NeMo-RL or standalone evaluator
              |
      Gym request / scheduler intent
              |
      OSWorld adapter <-> policy model
 model protocol, prompt, response parsing
              |
       runner / action horizon
              |
 OSWorld DesktopEnv -> VM -> task evaluator
              |
 semantic result + exact model-call evidence
              |
 runtime admission       trainer admission
       (Gym)             (training consumer)

The ownership boundaries are deliberate:

• Adapter: owns model wire format, history rendering, response/action parsing, and lossless parse-error facts. It does not fabricate FAIL, score a task, or decide max_steps.
• Runner: executes accepted actions, controls the horizon, invokes the evaluator after the final state transition, and records why execution stopped.
• Evaluator: is the sole authority for the OSWorld task score. Reaching max_steps does not rewrite its result.
• Runtime admission: Gym classifies execution/evaluation reliability and exposes the compatibility mask_sample view.
• Trainer admission: the consumer independently validates exact evidence and decides token/loss eligibility; it does not recompute the OSWorld score.

Logical rollout identity is separate from physical execution identity, so scheduler retries cannot silently merge two attempts. Non-idempotent /run transport calls are not retried in place; the scheduler may start a new, separately identified execution. rollout_purpose carries explicit scheduler intent without changing task, prompt, evaluator, or reward semantics, and legacy benchmark callers remain valid when it is absent.

History policy and model protocol are separate contracts. This PR includes fixed-window behavior and the hysteresis low/high-water policy used for 3-10-3 training. Each actual model-call boundary is preserved even when compaction changes the prompt prefix. Additional experimental policies remain in focused follow-ups rather than adding model-specific branches to the runner.

Main changes

• Adds OSWorld benchmark preparation, task conversion, model/environment probes, fixed-denominator metrics, launch/cleanup wrappers, and user-facing runtime documentation.
• Adds the OSWorld Responses API agent, runner registry, model/history contracts, exact trace envelope, semantic trajectory records, runtime outcome classification, and stable correlation IDs.
• Captures ordered model-call evidence including prompts, generated tokens, sampled logprobs, media identities, actions, environment transitions, evaluator result, and termination facts.
• Adds Gym Sandbox-backed OSWorld execution with local/remote Docker, dynamic service ports, /dev/kvm, immutable qcow2 identity, and run-scoped cleanup.
• Adds OpenSandbox Pool execution without exposing Pool implementation details to the model or trainer.
• Adds the OSWorld resources server, session-affine worker registration, capacity accounting, deployment identity, lifecycle management, and payload-free diagnostics.
• Adds managed runtime dependency checks and source-provenance gates while keeping changing Gym source mountable and deployment images, checkpoints, credentials, and cluster scripts outside the source PR.
• Adds focused unit and integration coverage for adapters, runner/evaluator ordering, admission, exact evidence, correlation, sandbox providers, resources-server lifecycle, CLI preparation, and runtime dependencies.

Standard preparation and execution

From a clean Linux checkout, prepare a five-task Nano Omni smoke run against a Docker/KVM environment:

uv lock --check
uv sync --frozen --extra dev --extra sandbox

python3 benchmarks/osworld/tools/probe_model_endpoint.py \
  --base-url http://MODEL_HOST:8000/v1 \
  --api-key local-vllm \
  --model SERVED_NANO_OMNI_MODEL \
  --image-count 3

bash benchmarks/osworld/tools/check_environment.sh \
  /absolute/path/to/Ubuntu.qcow2

cd benchmarks/osworld
python3 prepare.py \
  --profile nano_omni \
  --execution-backend gym_sandbox \
  --vm-path /absolute/path/to/Ubuntu.qcow2 \
  --policy-base-url http://MODEL_HOST:8000/v1 \
  --policy-model-name SERVED_NANO_OMNI_MODEL

export OSWORLD_RUN_ID=my-osworld-run
tools/start_control.sh /absolute/run/root

After the control log reports ready, run from a second terminal or supervisor:

cd benchmarks/osworld
tools/run_eval.sh /absolute/run/root

The benchmark README documents the equivalent OpenSandbox Pool flow, split-host Docker setup, full-population task generation, runtime dependency preparation, evidence locations, and run-scoped recovery.

Validation

• The branch is synchronized with current upstream main at 9d164bf38; feat(osworld): add exact multi-turn training support #2554 currently contains 85 changed files relative to that base.
• The complete focused current-head OSWorld, Resources Server, rollout/correlation, CLI/runtime, sandbox-provider, and vLLM-model regression suite passes.
• Ruff lint and format checks pass for every changed Python file; shell syntax and git diff --check pass.
• The PR-scope audit found no .gitignore, pre-commit, GitHub workflow, root dependency-policy, or lockfile changes relative to current main.
• Every feature/fix commit through 405e9537a carries the required DCO sign-off.
• End-to-end OpenSandbox evidence classified all 369 canonical tasks for Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16. It recorded 188 binary successes: 188/369 (50.95%) on the fixed denominator and 188/361 (52.08%) across the 361 evaluable tasks; the other eight were terminal Google Drive infrastructure failures rather than silently dropped rows.

Operational requirements and non-goals

• The default Docker image is x86_64 and requires Docker, read/write /dev/kvm, the verified qcow2, and sufficient host memory/disk. OpenSandbox instead requires a reachable management API and a compatible pre-provisioned KVM Pool.
• A run must record the exact Gym, OSWorld, model-protocol, history-policy, model/checkpoint, and runtime identities it actually used. The source contract declares compatible API ranges; a reproducibility record may pin the exact versions used by one experiment.
• This PR does not implement NeMo-RL advantage calculation, sequence-level loss masking, importance correction, replay buffering, checkpointing, or backward.
• It does not require unrelated Gym agents to implement exact trace, adopt OSWorld history policies, or use OSWorld admission semantics.
• It does not include AgentENV/E2B, sink-window, deployment images, model assets, credentials, site-specific Slurm scripts, or raw experiment artifacts; those remain in the follow-ups or deployment/recipe assets named above.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Glorf]

/claude review

[Glorf]

@ananthsub + @hemildesai could you take a look on the RL contract + sandbox logic respectively

[claude]

SHIP WITH CARE — the OSWorld env additions are solid and well-tested; my concern is a few shared-path changes riding along in an OSWorld-scoped PR.

Verifier / scoring correctness

• The new OSWorldResourcesServer.verify() (evaluate → close in finally) is correct and masks on evaluator throw. binary reward via score >= 1.0 matches the existing agent convention.
• Two shared-collector changes (rollout_collection.py) alter behavior for every agent, not just OSWorld — see the two inline RISK comments (mask_sample reclassification changing eval denominators + retry cost; unconditional retry_transport_errors=False). Neither is wrong for OSWorld; both should be an explicit, re-baselined decision for the other agents that share this loop, or scoped to OSWorld.

Async / operability — looks good

• session_manager.py wraps every blocking DesktopEnv call in asyncio.to_thread, bounds concurrency with a per-session asyncio.Lock + a manager _lock, and makes /step idempotent by operation_id (bounded to 128). Reaper task is cancelled cleanly on stop().
• server_utils.global_aiohttp_client_exit now routes close onto the owning loop rather than blindly asyncio.run() — good fix, covered by tests. _connectable_client_host wildcard→loopback mapping is sensible.
• Docker create reconciliation (_run_create_with_reconciliation) authenticates ownership via label+image before reuse and refuses to remove unrelated collisions — careful and correct.

API compatibility — preserved

• BaseRunRequest captures _ng_execution_id in a PrivateAttr via a wrap-validator; model_dump() shape is unchanged (asserted by tests). rollout_id_from_run / rollout_context / maybe_rollout_id_from_run_body retained as aliases. No public-surface breakage found.

Dependency hygiene

• resources_servers/osworld/pyproject.toml guards the heavy OSWorld/torchvision/OCR deps via uv override/exclude; _default_env_factory imports desktop_env lazily so the module stays importable without it. Intra-PR inconsistency worth a deliberate check: this file pins numpy<2 / opencv-python-headless~=4.8.1.78 while the agent installer moves to numpy>=2.1,<2.5 / OpenCV 4.10 — separate venvs so not a conflict, but confirm the resources-server venv actually resolves on py3.13.

Notes (author's call)

• Several new print(..., flush=True) diagnostic markers on the hot model/dispatch path (OSWORLD_MODEL_PURPOSE|, NEMO_GYM_VLLM_PROXY_PURPOSE|, per-dispatch /run summary). Payload-free and fine, but they're now unconditional stdout on every model call across all vLLM traffic, not just OSWorld.

No BLOCKERs. Recommend confirming the two shared-collector changes are intended for non-OSWorld agents before merge.

[JeffPengCoder]

Closing this original combined PR in favor of the scope-separated follow-ups:

• feat(osworld): isolate exact-trace and AgentENV environment support #3592 — OSWorld runtime, exact-trace contracts, and AgentENV environment support. Merged.
• fix(core): isolate component runtimes and preserve rollout identity #3591 — component runtime policy, rollout identity, and vLLM transport-log context. Merged.
• fix(sandbox): reconcile Docker creates and resolve E2B service endpoints #4822 — sandbox-provider fixes, E2B service endpoints, and embedded-worker transport support. Merged.
• fix(core): close shared aiohttp clients on their owner loop #4823 — shared aiohttp owner-loop lifecycle and wildcard-host handling. Still open.

The legacy compaction-session change was dropped from the core split in favor of #3368, which is also still open.

The remaining work continues in those focused PRs. The source branch is retained for provenance.